diff --git a/CHANGES.md b/CHANGES.md
index 548dfae9..25188793 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -11,11 +11,12 @@ Notable changes between versions.
#### AWS
-* Change Container Linux `os_image` default from `coreos-stable` to `flatcar-stable`
+* Change Container Linux `os_image` default from `coreos-stable` to `flatcar-stable` ([#702](https://github.com/poseidon/typhoon/pull/702))
#### Azure
-* Change Container Linux `os_image` default from `coreos-stable` to `flatcar-stable`
+* Add support for Fedora CoreOS ([#704](https://github.com/poseidon/typhoon/pull/704))
+* Change Container Linux `os_image` default from `coreos-stable` to `flatcar-stable` ([#702](https://github.com/poseidon/typhoon/pull/702))
#### Bare-Metal
@@ -23,11 +24,11 @@ Notable changes between versions.
#### Google
-* Change Container Linux `os_image` to be required. Container Linux users should upload a Flatcar Linux image and set it (**action required**)
+* Change Container Linux `os_image` to be required. Container Linux users should upload a Flatcar Linux image and set it (**action required**) ([#702](https://github.com/poseidon/typhoon/pull/702))
#### DigitalOcean
-* Change Container Linux `os_image` to be required. Container Linux users should upload a Flatcar Linux image and set it (**action required**)
+* Change Container Linux `os_image` to be required. Container Linux users should upload a Flatcar Linux image and set it (**action required**) ([#702](https://github.com/poseidon/typhoon/pull/702))
## v1.18.1
@@ -44,7 +45,7 @@ Notable changes between versions.
* Rename Container Linux `controller_clc_snippets` to `controller_snippets` for consistency ([#688](https://github.com/poseidon/typhoon/pull/688))
* Rename Container Linux `worker_clc_snippets` to `worker_snippets` for consistency
* Rename Container Linux `clc_snippets` (bare-metal) to `snippets` for consistency
-* Drop support for [gitRepo](https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo) volumes
+* Drop support for [gitRepo](https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo) volumes ([kubelet#3](https://github.com/poseidon/kubelet/pull/3))
#### Azure
diff --git a/README.md b/README.md
index 93520d59..863f5e04 100644
--- a/README.md
+++ b/README.md
@@ -26,6 +26,7 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
| Platform | Operating System | Terraform Module | Status |
|---------------|------------------|------------------|--------|
| AWS | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | stable |
+| Azure | Fedora CoreOS | [azure/fedora-coreos/kubernetes](azure/fedora-coreos/kubernetes) | alpha |
| Bare-Metal | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
| DigitalOcean | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | alpha |
| Google Cloud | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |
@@ -54,7 +55,7 @@ Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com
* [Docs](https://typhoon.psdn.io)
* Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
-* Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), [DigitalOcean](docs/fedora-coreos/digitalocean.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
+* Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Azure](docs/fedora-coreos/azure.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), [DigitalOcean](docs/fedora-coreos/digitalocean.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
* Flatcar Linux tutorials for [AWS](docs/cl/aws.md), [Azure](docs/cl/azure.md), [Bare-Metal](docs/cl/bare-metal.md), [DigitalOcean](docs/cl/digital-ocean.md), and [Google Cloud](docs/cl/google-cloud.md)
## Usage
diff --git a/azure/container-linux/kubernetes/variables.tf b/azure/container-linux/kubernetes/variables.tf
index 96eae93e..44827db4 100644
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@@ -60,7 +60,7 @@ variable "disk_size" {
variable "worker_priority" {
type = string
- description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
+ description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
default = "Regular"
}
diff --git a/azure/container-linux/kubernetes/workers/variables.tf b/azure/container-linux/kubernetes/workers/variables.tf
index 06ab9701..0ebd606f 100644
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@@ -52,7 +52,7 @@ variable "os_image" {
variable "priority" {
type = string
- description = "Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
+ description = "Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
default = "Regular"
}
diff --git a/azure/fedora-coreos/kubernetes/LICENSE b/azure/fedora-coreos/kubernetes/LICENSE
new file mode 100644
index 00000000..658b1c46
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/LICENSE
@@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Typhoon Authors
+Copyright (c) 2020 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
diff --git a/azure/fedora-coreos/kubernetes/README.md b/azure/fedora-coreos/kubernetes/README.md
new file mode 100644
index 00000000..e5ee3787
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/README.md
@@ -0,0 +1,23 @@
+# Typhoon 
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features 
+
+* Kubernetes v1.18.1 (upstream)
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
+* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/fedora-coreos/azure/).
+
diff --git a/azure/fedora-coreos/kubernetes/bootstrap.tf b/azure/fedora-coreos/kubernetes/bootstrap.tf
new file mode 100644
index 00000000..e22a3401
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@@ -0,0 +1,26 @@
+# Kubernetes assets (kubeconfig, manifests)
+module "bootstrap" {
+ source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+
+ cluster_name = var.cluster_name
+ api_servers = [format("%s.%s", var.cluster_name, var.dns_zone)]
+ etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
+ asset_dir = var.asset_dir
+
+ networking = var.networking
+
+ # only effective with Calico networking
+ # we should be able to use 1450 MTU, but in practice, 1410 was needed
+ network_encapsulation = "vxlan"
+ network_mtu = "1410"
+
+ pod_cidr = var.pod_cidr
+ service_cidr = var.service_cidr
+ cluster_domain_suffix = var.cluster_domain_suffix
+ enable_reporting = var.enable_reporting
+ enable_aggregation = var.enable_aggregation
+
+ # Fedora CoreOS
+ trusted_certs_dir = "/etc/pki/tls/certs"
+}
+
diff --git a/azure/fedora-coreos/kubernetes/controllers.tf b/azure/fedora-coreos/kubernetes/controllers.tf
new file mode 100644
index 00000000..b8299231
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/controllers.tf
@@ -0,0 +1,151 @@
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "azurerm_dns_a_record" "etcds" {
+ count = var.controller_count
+ resource_group_name = var.dns_zone_group
+
+ # DNS Zone name where record should be created
+ zone_name = var.dns_zone
+
+ # DNS record
+ name = format("%s-etcd%d", var.cluster_name, count.index)
+ ttl = 300
+
+ # private IPv4 address for etcd
+ records = [azurerm_network_interface.controllers.*.private_ip_address[count.index]]
+}
+
+# Controller availability set to spread controllers
+resource "azurerm_availability_set" "controllers" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-controllers"
+ location = var.region
+ platform_fault_domain_count = 2
+ platform_update_domain_count = 4
+ managed = true
+}
+
+# Controller instances
+resource "azurerm_linux_virtual_machine" "controllers" {
+ count = var.controller_count
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-controller-${count.index}"
+ location = var.region
+ availability_set_id = azurerm_availability_set.controllers.id
+
+ size = var.controller_type
+ custom_data = base64encode(data.ct_config.controller-ignitions.*.rendered[count.index])
+
+ # storage
+ source_image_id = var.os_image
+ os_disk {
+ name = "${var.cluster_name}-controller-${count.index}"
+ caching = "None"
+ disk_size_gb = var.disk_size
+ storage_account_type = "Premium_LRS"
+ }
+
+ # network
+ network_interface_ids = [
+ azurerm_network_interface.controllers.*.id[count.index]
+ ]
+
+ # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
+ admin_username = "core"
+ admin_ssh_key {
+ username = "core"
+ public_key = var.ssh_authorized_key
+ }
+
+ lifecycle {
+ ignore_changes = [
+ os_disk,
+ custom_data,
+ ]
+ }
+}
+
+# Controller public IPv4 addresses
+resource "azurerm_public_ip" "controllers" {
+ count = var.controller_count
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-controller-${count.index}"
+ location = azurerm_resource_group.cluster.location
+ sku = "Standard"
+ allocation_method = "Static"
+}
+
+# Controller NICs with public and private IPv4
+resource "azurerm_network_interface" "controllers" {
+ count = var.controller_count
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-controller-${count.index}"
+ location = azurerm_resource_group.cluster.location
+
+ ip_configuration {
+ name = "ip0"
+ subnet_id = azurerm_subnet.controller.id
+ private_ip_address_allocation = "Dynamic"
+ # instance public IPv4
+ public_ip_address_id = azurerm_public_ip.controllers.*.id[count.index]
+ }
+}
+
+# Associate controller network interface with controller security group
+resource "azurerm_network_interface_security_group_association" "controllers" {
+ count = var.controller_count
+
+ network_interface_id = azurerm_network_interface.controllers[count.index].id
+ network_security_group_id = azurerm_network_security_group.controller.id
+}
+
+# Associate controller network interface with controller backend address pool
+resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+ count = var.controller_count
+
+ network_interface_id = azurerm_network_interface.controllers[count.index].id
+ ip_configuration_name = "ip0"
+ backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+}
+
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+ count = var.controller_count
+ content = data.template_file.controller-configs.*.rendered[count.index]
+ pretty_print = false
+ snippets = var.controller_snippets
+}
+
+# Controller Fedora CoreOS configs
+data "template_file" "controller-configs" {
+ count = var.controller_count
+
+ template = file("${path.module}/fcc/controller.yaml")
+
+ vars = {
+ # Cannot use cyclic dependencies on controllers or their DNS records
+ etcd_name = "etcd${count.index}"
+ etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+ # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+ etcd_initial_cluster = join(",", data.template_file.etcds.*.rendered)
+ kubeconfig = indent(10, module.bootstrap.kubeconfig-kubelet)
+ ssh_authorized_key = var.ssh_authorized_key
+ cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+ cluster_domain_suffix = var.cluster_domain_suffix
+ }
+}
+
+data "template_file" "etcds" {
+ count = var.controller_count
+ template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+ vars = {
+ index = count.index
+ cluster_name = var.cluster_name
+ dns_zone = var.dns_zone
+ }
+}
+
diff --git a/azure/fedora-coreos/kubernetes/fcc/controller.yaml b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
new file mode 100644
index 00000000..06c8d1d2
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@@ -0,0 +1,213 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+ units:
+ - name: etcd-member.service
+ enabled: true
+ contents: |
+ [Unit]
+ Description=etcd (System Container)
+ Documentation=https://github.com/coreos/etcd
+ Wants=network-online.target network.target
+ After=network-online.target
+ [Service]
+ # https://github.com/opencontainers/runc/pull/1807
+ # Type=notify
+ # NotifyAccess=exec
+ Type=exec
+ Restart=on-failure
+ RestartSec=10s
+ TimeoutStartSec=0
+ LimitNOFILE=40000
+ ExecStartPre=/bin/mkdir -p /var/lib/etcd
+ ExecStartPre=-/usr/bin/podman rm etcd
+ #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
+ ExecStart=/usr/bin/podman run --name etcd \
+ --env-file /etc/etcd/etcd.env \
+ --network host \
+ --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
+ --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
+ quay.io/coreos/etcd:v3.4.7
+ ExecStop=/usr/bin/podman stop etcd
+ [Install]
+ WantedBy=multi-user.target
+ - name: docker.service
+ enabled: true
+ - name: wait-for-dns.service
+ enabled: true
+ contents: |
+ [Unit]
+ Description=Wait for DNS entries
+ Before=kubelet.service
+ [Service]
+ Type=oneshot
+ RemainAfterExit=true
+ ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+ [Install]
+ RequiredBy=kubelet.service
+ RequiredBy=etcd-member.service
+ - name: kubelet.service
+ enabled: true
+ contents: |
+ [Unit]
+ Description=Kubelet via Hyperkube (System Container)
+ Wants=rpc-statd.service
+ [Service]
+ ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+ ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+ ExecStartPre=/bin/mkdir -p /opt/cni/bin
+ ExecStartPre=/bin/mkdir -p /var/lib/calico
+ ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+ ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+ ExecStartPre=-/usr/bin/podman rm kubelet
+ ExecStart=/usr/bin/podman run --name kubelet \
+ --privileged \
+ --pid host \
+ --network host \
+ --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+ --volume /usr/lib/os-release:/etc/os-release:ro \
+ --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+ --volume /lib/modules:/lib/modules:ro \
+ --volume /run:/run \
+ --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+ --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+ --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+ --volume /var/lib/calico:/var/lib/calico:ro \
+ --volume /var/lib/docker:/var/lib/docker \
+ --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+ --volume /var/log:/var/log \
+ --volume /var/run/lock:/var/run/lock:z \
+ --volume /opt/cni/bin:/opt/cni/bin:z \
+ quay.io/poseidon/kubelet:v1.18.1 \
+ --anonymous-auth=false \
+ --authentication-token-webhook \
+ --authorization-mode=Webhook \
+ --cgroup-driver=systemd \
+ --cgroups-per-qos=true \
+ --enforce-node-allocatable=pods \
+ --client-ca-file=/etc/kubernetes/ca.crt \
+ --cluster_dns=${cluster_dns_service_ip} \
+ --cluster_domain=${cluster_domain_suffix} \
+ --cni-conf-dir=/etc/kubernetes/cni/net.d \
+ --exit-on-lock-contention \
+ --healthz-port=0 \
+ --kubeconfig=/etc/kubernetes/kubeconfig \
+ --lock-file=/var/run/lock/kubelet.lock \
+ --network-plugin=cni \
+ --node-labels=node.kubernetes.io/master \
+ --node-labels=node.kubernetes.io/controller="true" \
+ --pod-manifest-path=/etc/kubernetes/manifests \
+ --read-only-port=0 \
+ --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+ --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+ ExecStop=-/usr/bin/podman stop kubelet
+ Delegate=yes
+ Restart=always
+ RestartSec=10
+ [Install]
+ WantedBy=multi-user.target
+ - name: bootstrap.service
+ contents: |
+ [Unit]
+ Description=Kubernetes control plane
+ ConditionPathExists=!/opt/bootstrap/bootstrap.done
+ [Service]
+ Type=oneshot
+ RemainAfterExit=true
+ WorkingDirectory=/opt/bootstrap
+ ExecStartPre=-/usr/bin/podman rm bootstrap
+ ExecStart=/usr/bin/podman run --name bootstrap \
+ --network host \
+ --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+ --volume /opt/bootstrap/assets:/assets:ro,Z \
+ --volume /opt/bootstrap/apply:/apply:ro,Z \
+ --entrypoint=/apply \
+ quay.io/poseidon/kubelet:v1.18.1
+ ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+ ExecStartPost=-/usr/bin/podman stop bootstrap
+storage:
+ directories:
+ - path: /etc/kubernetes
+ - path: /opt/bootstrap
+ files:
+ - path: /etc/kubernetes/kubeconfig
+ mode: 0644
+ contents:
+ inline: |
+ ${kubeconfig}
+ - path: /opt/bootstrap/layout
+ mode: 0544
+ contents:
+ inline: |
+ #!/bin/bash -e
+ mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+ awk '/#####/ {filename=$2; next} {print > filename}' assets
+ mkdir -p /etc/ssl/etcd/etcd
+ mkdir -p /etc/kubernetes/bootstrap-secrets
+ mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+ mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+ chown -R etcd:etcd /etc/ssl/etcd
+ chmod -R 500 /etc/ssl/etcd
+ mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+ mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+ sudo mkdir -p /etc/kubernetes/manifests
+ sudo mv static-manifests/* /etc/kubernetes/manifests/
+ sudo mkdir -p /opt/bootstrap/assets
+ sudo mv manifests /opt/bootstrap/assets/manifests
+ sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+ rm -rf assets auth static-manifests tls manifests-networking
+ - path: /opt/bootstrap/apply
+ mode: 0544
+ contents:
+ inline: |
+ #!/bin/bash -e
+ export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+ until kubectl version; do
+ echo "Waiting for static pod control plane"
+ sleep 5
+ done
+ until kubectl apply -f /assets/manifests -R; do
+ echo "Retry applying manifests"
+ sleep 5
+ done
+ - path: /etc/sysctl.d/max-user-watches.conf
+ contents:
+ inline: |
+ fs.inotify.max_user_watches=16184
+ - path: /etc/systemd/system.conf.d/accounting.conf
+ contents:
+ inline: |
+ [Manager]
+ DefaultCPUAccounting=yes
+ DefaultMemoryAccounting=yes
+ DefaultBlockIOAccounting=yes
+ - path: /etc/etcd/etcd.env
+ mode: 0644
+ contents:
+ inline: |
+ # TODO: Use a systemd dropin once podman v1.4.5 is avail.
+ NOTIFY_SOCKET=/run/systemd/notify
+ ETCD_NAME=${etcd_name}
+ ETCD_DATA_DIR=/var/lib/etcd
+ ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+ ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+ ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+ ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+ ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+ ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+ ETCD_STRICT_RECONFIG_CHECK=true
+ ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+ ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+ ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+ ETCD_CLIENT_CERT_AUTH=true
+ ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+ ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+ ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+ ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+ users:
+ - name: core
+ ssh_authorized_keys:
+ - ${ssh_authorized_key}
+
diff --git a/azure/fedora-coreos/kubernetes/lb.tf b/azure/fedora-coreos/kubernetes/lb.tf
new file mode 100644
index 00000000..ef924759
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/lb.tf
@@ -0,0 +1,161 @@
+# DNS record for the apiserver load balancer
+resource "azurerm_dns_a_record" "apiserver" {
+ resource_group_name = var.dns_zone_group
+
+ # DNS Zone name where record should be created
+ zone_name = var.dns_zone
+
+ # DNS record
+ name = var.cluster_name
+ ttl = 300
+
+ # IPv4 address of apiserver load balancer
+ records = [azurerm_public_ip.apiserver-ipv4.ip_address]
+}
+
+# Static IPv4 address for the apiserver frontend
+resource "azurerm_public_ip" "apiserver-ipv4" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-apiserver-ipv4"
+ location = var.region
+ sku = "Standard"
+ allocation_method = "Static"
+}
+
+# Static IPv4 address for the ingress frontend
+resource "azurerm_public_ip" "ingress-ipv4" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-ingress-ipv4"
+ location = var.region
+ sku = "Standard"
+ allocation_method = "Static"
+}
+
+# Network Load Balancer for apiservers and ingress
+resource "azurerm_lb" "cluster" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = var.cluster_name
+ location = var.region
+ sku = "Standard"
+
+ frontend_ip_configuration {
+ name = "apiserver"
+ public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
+ }
+
+ frontend_ip_configuration {
+ name = "ingress"
+ public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
+ }
+}
+
+resource "azurerm_lb_rule" "apiserver" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "apiserver"
+ loadbalancer_id = azurerm_lb.cluster.id
+ frontend_ip_configuration_name = "apiserver"
+
+ protocol = "Tcp"
+ frontend_port = 6443
+ backend_port = 6443
+ backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+ probe_id = azurerm_lb_probe.apiserver.id
+}
+
+resource "azurerm_lb_rule" "ingress-http" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "ingress-http"
+ loadbalancer_id = azurerm_lb.cluster.id
+ frontend_ip_configuration_name = "ingress"
+ disable_outbound_snat = true
+
+ protocol = "Tcp"
+ frontend_port = 80
+ backend_port = 80
+ backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+ probe_id = azurerm_lb_probe.ingress.id
+}
+
+resource "azurerm_lb_rule" "ingress-https" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "ingress-https"
+ loadbalancer_id = azurerm_lb.cluster.id
+ frontend_ip_configuration_name = "ingress"
+ disable_outbound_snat = true
+
+ protocol = "Tcp"
+ frontend_port = 443
+ backend_port = 443
+ backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+ probe_id = azurerm_lb_probe.ingress.id
+}
+
+# Worker outbound TCP/UDP SNAT
+resource "azurerm_lb_outbound_rule" "worker-outbound" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "worker"
+ loadbalancer_id = azurerm_lb.cluster.id
+ frontend_ip_configuration {
+ name = "ingress"
+ }
+
+ protocol = "All"
+ backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+}
+
+# Address pool of controllers
+resource "azurerm_lb_backend_address_pool" "controller" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "controller"
+ loadbalancer_id = azurerm_lb.cluster.id
+}
+
+# Address pool of workers
+resource "azurerm_lb_backend_address_pool" "worker" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "worker"
+ loadbalancer_id = azurerm_lb.cluster.id
+}
+
+# Health checks / probes
+
+# TCP health check for apiserver
+resource "azurerm_lb_probe" "apiserver" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "apiserver"
+ loadbalancer_id = azurerm_lb.cluster.id
+ protocol = "Tcp"
+ port = 6443
+
+ # unhealthy threshold
+ number_of_probes = 3
+
+ interval_in_seconds = 5
+}
+
+# HTTP health check for ingress
+resource "azurerm_lb_probe" "ingress" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "ingress"
+ loadbalancer_id = azurerm_lb.cluster.id
+ protocol = "Http"
+ port = 10254
+ request_path = "/healthz"
+
+ # unhealthy threshold
+ number_of_probes = 3
+
+ interval_in_seconds = 5
+}
+
diff --git a/azure/fedora-coreos/kubernetes/network.tf b/azure/fedora-coreos/kubernetes/network.tf
new file mode 100644
index 00000000..ea92a5a7
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/network.tf
@@ -0,0 +1,44 @@
+# Organize cluster into a resource group
+resource "azurerm_resource_group" "cluster" {
+ name = var.cluster_name
+ location = var.region
+}
+
+resource "azurerm_virtual_network" "network" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = var.cluster_name
+ location = azurerm_resource_group.cluster.location
+ address_space = [var.host_cidr]
+}
+
+# Subnets - separate subnets for controller and workers because Azure
+# network security groups are based on IPv4 CIDR rather than instance
+# tags like GCP or security group membership like AWS
+
+resource "azurerm_subnet" "controller" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "controller"
+ virtual_network_name = azurerm_virtual_network.network.name
+ address_prefix = cidrsubnet(var.host_cidr, 1, 0)
+}
+
+resource "azurerm_subnet_network_security_group_association" "controller" {
+ subnet_id = azurerm_subnet.controller.id
+ network_security_group_id = azurerm_network_security_group.controller.id
+}
+
+resource "azurerm_subnet" "worker" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "worker"
+ virtual_network_name = azurerm_virtual_network.network.name
+ address_prefix = cidrsubnet(var.host_cidr, 1, 1)
+}
+
+resource "azurerm_subnet_network_security_group_association" "worker" {
+ subnet_id = azurerm_subnet.worker.id
+ network_security_group_id = azurerm_network_security_group.worker.id
+}
+
diff --git a/azure/fedora-coreos/kubernetes/outputs.tf b/azure/fedora-coreos/kubernetes/outputs.tf
new file mode 100644
index 00000000..fe5eaffb
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/outputs.tf
@@ -0,0 +1,59 @@
+output "kubeconfig-admin" {
+ value = module.bootstrap.kubeconfig-admin
+}
+
+# Outputs for Kubernetes Ingress
+
+output "ingress_static_ipv4" {
+ value = azurerm_public_ip.ingress-ipv4.ip_address
+ description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
+# Outputs for worker pools
+
+output "region" {
+ value = azurerm_resource_group.cluster.location
+}
+
+output "resource_group_name" {
+ value = azurerm_resource_group.cluster.name
+}
+
+output "resource_group_id" {
+ value = azurerm_resource_group.cluster.id
+}
+
+output "subnet_id" {
+ value = azurerm_subnet.worker.id
+}
+
+output "security_group_id" {
+ value = azurerm_network_security_group.worker.id
+}
+
+output "kubeconfig" {
+ value = module.bootstrap.kubeconfig-kubelet
+}
+
+# Outputs for custom firewalling
+
+output "worker_security_group_name" {
+ value = azurerm_network_security_group.worker.name
+}
+
+output "worker_address_prefix" {
+ description = "Worker network subnet CIDR address (for source/destination)"
+ value = azurerm_subnet.worker.address_prefix
+}
+
+# Outputs for custom load balancing
+
+output "loadbalancer_id" {
+ description = "ID of the cluster load balancer"
+ value = azurerm_lb.cluster.id
+}
+
+output "backend_address_pool_id" {
+ description = "ID of the worker backend address pool"
+ value = azurerm_lb_backend_address_pool.worker.id
+}
diff --git a/azure/fedora-coreos/kubernetes/security.tf b/azure/fedora-coreos/kubernetes/security.tf
new file mode 100644
index 00000000..feb6fef5
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/security.tf
@@ -0,0 +1,336 @@
+# Controller security group
+
+resource "azurerm_network_security_group" "controller" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-controller"
+ location = azurerm_resource_group.cluster.location
+}
+
+resource "azurerm_network_security_rule" "controller-ssh" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-ssh"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2000"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "22"
+ source_address_prefix = "*"
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+resource "azurerm_network_security_rule" "controller-etcd" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-etcd"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2005"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "2379-2380"
+ source_address_prefix = azurerm_subnet.controller.address_prefix
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape etcd metrics
+resource "azurerm_network_security_rule" "controller-etcd-metrics" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-etcd-metrics"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2010"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "2381"
+ source_address_prefix = azurerm_subnet.worker.address_prefix
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape kube-proxy metrics
+resource "azurerm_network_security_rule" "controller-kube-proxy" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-kube-proxy-metrics"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2011"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "10249"
+ source_address_prefix = azurerm_subnet.worker.address_prefix
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
+resource "azurerm_network_security_rule" "controller-kube-metrics" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-kube-metrics"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2012"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "10251-10252"
+ source_address_prefix = azurerm_subnet.worker.address_prefix
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+resource "azurerm_network_security_rule" "controller-apiserver" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-apiserver"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2015"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "6443"
+ source_address_prefix = "*"
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+resource "azurerm_network_security_rule" "controller-vxlan" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-vxlan"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2020"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Udp"
+ source_port_range = "*"
+ destination_port_range = "4789"
+ source_address_prefixes = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "controller-node-exporter" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-node-exporter"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2025"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "9100"
+ source_address_prefix = azurerm_subnet.worker.address_prefix
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "controller-kubelet" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-kubelet"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "2030"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "10250"
+
+ # allow Prometheus to scrape kubelet metrics too
+ source_address_prefixes = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+ destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-loadbalancer"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "3000"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "*"
+ source_port_range = "*"
+ destination_port_range = "*"
+ source_address_prefix = "AzureLoadBalancer"
+ destination_address_prefix = "*"
+}
+
+resource "azurerm_network_security_rule" "controller-deny-all" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "deny-all"
+ network_security_group_name = azurerm_network_security_group.controller.name
+ priority = "3005"
+ access = "Deny"
+ direction = "Inbound"
+ protocol = "*"
+ source_port_range = "*"
+ destination_port_range = "*"
+ source_address_prefix = "*"
+ destination_address_prefix = "*"
+}
+
+# Worker security group
+
+resource "azurerm_network_security_group" "worker" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "${var.cluster_name}-worker"
+ location = azurerm_resource_group.cluster.location
+}
+
+resource "azurerm_network_security_rule" "worker-ssh" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-ssh"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2000"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "22"
+ source_address_prefix = azurerm_subnet.controller.address_prefix
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+resource "azurerm_network_security_rule" "worker-http" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-http"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2005"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "80"
+ source_address_prefix = "*"
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+resource "azurerm_network_security_rule" "worker-https" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-https"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2010"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "443"
+ source_address_prefix = "*"
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+resource "azurerm_network_security_rule" "worker-vxlan" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-vxlan"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2015"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Udp"
+ source_port_range = "*"
+ destination_port_range = "4789"
+ source_address_prefixes = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "worker-node-exporter" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-node-exporter"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2020"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "9100"
+ source_address_prefix = azurerm_subnet.worker.address_prefix
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+# Allow Prometheus to scrape kube-proxy
+resource "azurerm_network_security_rule" "worker-kube-proxy" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-kube-proxy"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2024"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "10249"
+ source_address_prefix = azurerm_subnet.worker.address_prefix
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "worker-kubelet" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-kubelet"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "2025"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "Tcp"
+ source_port_range = "*"
+ destination_port_range = "10250"
+
+ # allow Prometheus to scrape kubelet metrics too
+ source_address_prefixes = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+ destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "allow-loadbalancer"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "3000"
+ access = "Allow"
+ direction = "Inbound"
+ protocol = "*"
+ source_port_range = "*"
+ destination_port_range = "*"
+ source_address_prefix = "AzureLoadBalancer"
+ destination_address_prefix = "*"
+}
+
+resource "azurerm_network_security_rule" "worker-deny-all" {
+ resource_group_name = azurerm_resource_group.cluster.name
+
+ name = "deny-all"
+ network_security_group_name = azurerm_network_security_group.worker.name
+ priority = "3005"
+ access = "Deny"
+ direction = "Inbound"
+ protocol = "*"
+ source_port_range = "*"
+ destination_port_range = "*"
+ source_address_prefix = "*"
+ destination_address_prefix = "*"
+}
+
diff --git a/azure/fedora-coreos/kubernetes/ssh.tf b/azure/fedora-coreos/kubernetes/ssh.tf
new file mode 100644
index 00000000..04f54393
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/ssh.tf
@@ -0,0 +1,59 @@
+locals {
+ # format assets for distribution
+ assets_bundle = [
+ # header with the unpack location
+ for key, value in module.bootstrap.assets_dist :
+ format("##### %s\n%s", key, value)
+ ]
+}
+
+# Secure copy assets to controllers.
+resource "null_resource" "copy-controller-secrets" {
+ count = var.controller_count
+
+ depends_on = [
+ module.bootstrap,
+ azurerm_linux_virtual_machine.controllers
+ ]
+
+ connection {
+ type = "ssh"
+ host = azurerm_public_ip.controllers.*.ip_address[count.index]
+ user = "core"
+ timeout = "15m"
+ }
+
+ provisioner "file" {
+ content = join("\n", local.assets_bundle)
+ destination = "$HOME/assets"
+ }
+
+ provisioner "remote-exec" {
+ inline = [
+ "sudo /opt/bootstrap/layout",
+ ]
+ }
+}
+
+# Connect to a controller to perform one-time cluster bootstrap.
+resource "null_resource" "bootstrap" {
+ depends_on = [
+ null_resource.copy-controller-secrets,
+ module.workers,
+ azurerm_dns_a_record.apiserver,
+ ]
+
+ connection {
+ type = "ssh"
+ host = azurerm_public_ip.controllers.*.ip_address[0]
+ user = "core"
+ timeout = "15m"
+ }
+
+ provisioner "remote-exec" {
+ inline = [
+ "sudo systemctl start bootstrap",
+ ]
+ }
+}
+
diff --git a/azure/fedora-coreos/kubernetes/variables.tf b/azure/fedora-coreos/kubernetes/variables.tf
new file mode 100644
index 00000000..1bff1e7e
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@@ -0,0 +1,143 @@
+variable "cluster_name" {
+ type = string
+ description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# Azure
+
+variable "region" {
+ type = string
+ description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+}
+
+variable "dns_zone" {
+ type = string
+ description = "Azure DNS Zone (e.g. azure.example.com)"
+}
+
+variable "dns_zone_group" {
+ type = string
+ description = "Resource group where the Azure DNS Zone resides (e.g. global)"
+}
+
+# instances
+
+variable "controller_count" {
+ type = number
+ description = "Number of controllers (i.e. masters)"
+ default = 1
+}
+
+variable "worker_count" {
+ type = number
+ description = "Number of workers"
+ default = 1
+}
+
+variable "controller_type" {
+ type = string
+ description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
+ default = "Standard_B2s"
+}
+
+variable "worker_type" {
+ type = string
+ description = "Machine type for workers (see `az vm list-skus --location centralus`)"
+ default = "Standard_DS1_v2"
+}
+
+variable "os_image" {
+ type = string
+ description = "Fedora CoreOS image for instances"
+}
+
+variable "disk_size" {
+ type = number
+ description = "Size of the disk in GB"
+ default = 40
+}
+
+variable "worker_priority" {
+ type = string
+ description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
+ default = "Regular"
+}
+
+variable "controller_snippets" {
+ type = list(string)
+ description = "Controller Fedora CoreOS Config snippets"
+ default = []
+}
+
+variable "worker_snippets" {
+ type = list(string)
+ description = "Worker Fedora CoreOS Config snippets"
+ default = []
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+ type = string
+ description = "SSH public key for user 'core'"
+}
+
+variable "networking" {
+ type = string
+ description = "Choice of networking provider (flannel or calico)"
+ default = "calico"
+}
+
+variable "host_cidr" {
+ type = string
+ description = "CIDR IPv4 range to assign to instances"
+ default = "10.0.0.0/16"
+}
+
+variable "pod_cidr" {
+ type = string
+ description = "CIDR IPv4 range to assign Kubernetes pods"
+ default = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+ type = string
+ description = < /dev/null; do sleep 1; done'
+ [Install]
+ RequiredBy=kubelet.service
+ - name: kubelet.service
+ enabled: true
+ contents: |
+ [Unit]
+ Description=Kubelet via Hyperkube (System Container)
+ Wants=rpc-statd.service
+ [Service]
+ ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+ ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+ ExecStartPre=/bin/mkdir -p /opt/cni/bin
+ ExecStartPre=/bin/mkdir -p /var/lib/calico
+ ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+ ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+ ExecStartPre=-/usr/bin/podman rm kubelet
+ ExecStart=/usr/bin/podman run --name kubelet \
+ --privileged \
+ --pid host \
+ --network host \
+ --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+ --volume /usr/lib/os-release:/etc/os-release:ro \
+ --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+ --volume /lib/modules:/lib/modules:ro \
+ --volume /run:/run \
+ --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+ --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+ --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+ --volume /var/lib/calico:/var/lib/calico:ro \
+ --volume /var/lib/docker:/var/lib/docker \
+ --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+ --volume /var/log:/var/log \
+ --volume /var/run/lock:/var/run/lock:z \
+ --volume /opt/cni/bin:/opt/cni/bin:z \
+ quay.io/poseidon/kubelet:v1.18.1 \
+ --anonymous-auth=false \
+ --authentication-token-webhook \
+ --authorization-mode=Webhook \
+ --cgroup-driver=systemd \
+ --cgroups-per-qos=true \
+ --enforce-node-allocatable=pods \
+ --client-ca-file=/etc/kubernetes/ca.crt \
+ --cluster_dns=${cluster_dns_service_ip} \
+ --cluster_domain=${cluster_domain_suffix} \
+ --cni-conf-dir=/etc/kubernetes/cni/net.d \
+ --exit-on-lock-contention \
+ --healthz-port=0 \
+ --kubeconfig=/etc/kubernetes/kubeconfig \
+ --lock-file=/var/run/lock/kubelet.lock \
+ --network-plugin=cni \
+ --node-labels=node.kubernetes.io/node \
+ %{~ for label in split(",", node_labels) ~}
+ --node-labels=${label} \
+ %{~ endfor ~}
+ --pod-manifest-path=/etc/kubernetes/manifests \
+ --read-only-port=0 \
+ --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+ ExecStop=-/usr/bin/podman stop kubelet
+ Delegate=yes
+ Restart=always
+ RestartSec=10
+ [Install]
+ WantedBy=multi-user.target
+ - name: delete-node.service
+ enabled: true
+ contents: |
+ [Unit]
+ Description=Delete Kubernetes node on shutdown
+ [Service]
+ Type=oneshot
+ RemainAfterExit=true
+ ExecStart=/bin/true
+ ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+ [Install]
+ WantedBy=multi-user.target
+storage:
+ directories:
+ - path: /etc/kubernetes
+ files:
+ - path: /etc/kubernetes/kubeconfig
+ mode: 0644
+ contents:
+ inline: |
+ ${kubeconfig}
+ - path: /etc/sysctl.d/max-user-watches.conf
+ contents:
+ inline: |
+ fs.inotify.max_user_watches=16184
+ - path: /etc/systemd/system.conf.d/accounting.conf
+ contents:
+ inline: |
+ [Manager]
+ DefaultCPUAccounting=yes
+ DefaultMemoryAccounting=yes
+ DefaultBlockIOAccounting=yes
+passwd:
+ users:
+ - name: core
+ ssh_authorized_keys:
+ - ${ssh_authorized_key}
+
+
diff --git a/azure/fedora-coreos/kubernetes/workers/variables.tf b/azure/fedora-coreos/kubernetes/workers/variables.tf
new file mode 100644
index 00000000..0e12fd08
--- /dev/null
+++ b/azure/fedora-coreos/kubernetes/workers/variables.tf
@@ -0,0 +1,98 @@
+variable "name" {
+ type = string
+ description = "Unique name for the worker pool"
+}
+
+# Azure
+
+variable "region" {
+ type = string
+ description = "Must be set to the Azure Region of cluster"
+}
+
+variable "resource_group_name" {
+ type = string
+ description = "Must be set to the resource group name of cluster"
+}
+
+variable "subnet_id" {
+ type = string
+ description = "Must be set to the `worker_subnet_id` output by cluster"
+}
+
+variable "security_group_id" {
+ type = string
+ description = "Must be set to the `worker_security_group_id` output by cluster"
+}
+
+variable "backend_address_pool_id" {
+ type = string
+ description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
+}
+
+# instances
+
+variable "worker_count" {
+ type = number
+ description = "Number of instances"
+ default = 1
+}
+
+variable "vm_type" {
+ type = string
+ description = "Machine type for instances (see `az vm list-skus --location centralus`)"
+ default = "Standard_DS1_v2"
+}
+
+variable "os_image" {
+ type = string
+ description = "Fedora CoreOS image for instances"
+}
+
+variable "priority" {
+ type = string
+ description = "Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
+ default = "Regular"
+}
+
+variable "snippets" {
+ type = list(string)
+ description = "Fedora CoreOS Config snippets"
+ default = []
+}
+
+# configuration
+
+variable "kubeconfig" {
+ type = string
+ description = "Must be set to `kubeconfig` output by cluster"
+}
+
+variable "ssh_authorized_key" {
+ type = string
+ description = "SSH public key for user 'core'"
+}
+
+variable "service_cidr" {
+ type = string
+ description = < 24m v1.18.1
+ramius-worker-000001 Ready 25m v1.18.1
+ramius-worker-000002 Ready 24m v1.18.1
+```
+
+List the pods.
+
+```
+$ kubectl get pods --all-namespaces
+NAMESPACE NAME READY STATUS RESTARTS AGE
+kube-system coredns-7c6fbb4f4b-b6qzx 1/1 Running 0 26m
+kube-system coredns-7c6fbb4f4b-j2k3d 1/1 Running 0 26m
+kube-system calico-node-1m5bf 2/2 Running 0 26m
+kube-system calico-node-7jmr1 2/2 Running 0 26m
+kube-system calico-node-bknc8 2/2 Running 0 26m
+kube-system kube-apiserver-ramius-controller-0 1/1 Running 0 26m
+kube-system kube-controller-manager-ramius-controller-0 1/1 Running 0 26m
+kube-system kube-proxy-j4vpq 1/1 Running 0 26m
+kube-system kube-proxy-jxr5d 1/1 Running 0 26m
+kube-system kube-proxy-lbdw5 1/1 Running 0 26m
+kube-system kube-scheduler-ramius-controller-0 1/1 Running 0 26m
+```
+
+## Going Further
+
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
+
+## Variables
+
+Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/fedora-coreos/kubernetes/variables.tf) source.
+
+### Required
+
+| Name | Description | Example |
+|:-----|:------------|:--------|
+| cluster_name | Unique cluster name (prepended to dns_zone) | "ramius" |
+| region | Azure region | "centralus" |
+| dns_zone | Azure DNS zone | "azure.example.com" |
+| dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
+| os_image | Fedora CoreOS image for instances | "/subscriptions/..../custom-image" |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+
+!!! tip
+ Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
+
+#### DNS Zone
+
+Clusters create a DNS A record `${cluster_name}.${dns_zone}` to resolve a load balancer backed by controller instances. This FQDN is used by workers and `kubectl` to access the apiserver(s). In this example, the cluster's apiserver would be accessible at `ramius.azure.example.com`.
+
+You'll need a registered domain name or delegated subdomain on Azure DNS. You can set this up once and create many clusters with unique names.
+
+```tf
+# Azure resource group for DNS zone
+resource "azurerm_resource_group" "global" {
+ name = "global"
+ location = "centralus"
+}
+
+# DNS zone for clusters
+resource "azurerm_dns_zone" "clusters" {
+ resource_group_name = azurerm_resource_group.global.name
+
+ name = "azure.example.com"
+ zone_type = "Public"
+}
+```
+
+Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource group with `"azurerm_resource_group.global.name`.
+
+!!! tip ""
+ If you have an existing domain name with a zone file elsewhere, just delegate a subdomain that can be managed on Azure DNS (e.g. azure.mydomain.com) and [update nameservers](https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns).
+
+### Optional
+
+| Name | Description | Default | Example |
+|:-----|:------------|:--------|:--------|
+| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
+| worker_count | Number of workers | 1 | 3 |
+| controller_type | Machine type for controllers | "Standard_B2s" | See below |
+| worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
+| disk_size | Size of the disk in GB | 40 | 100 |
+| worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
+| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
+| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
+| networking | Choice of networking provider | "calico" | "flannel" or "calico" |
+| host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
+| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
+| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| worker_node_labels | List of initial worker node labels | [] | ["worker-pool=default"] |
+
+Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
+
+!!! warning
+ Unlike AWS and GCP, Azure requires its *virtual* networks to have non-overlapping IPv4 CIDRs (yeah, go figure). Instead of each cluster just using `10.0.0.0/16` for instances, each Azure cluster's `host_cidr` must be non-overlapping (e.g. 10.0.0.0/20 for the 1st cluster, 10.0.16.0/20 for the 2nd cluster, etc).
+
+!!! warning
+ Do not choose a `controller_type` smaller than `Standard_B2s`. Smaller instances are not sufficient for running a controller.
+
+#### Spot Priority
+
+Add `worker_priority=Spot` to use [Spot Priority](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/spot-vms) workers that run on Azure's surplus capacity at lower cost, but with the tradeoff that they can be deallocated at random. Spot priority VMs are Azure's analog to AWS spot instances or GCP premptible instances.
diff --git a/docs/fedora-coreos/digitalocean.md b/docs/fedora-coreos/digitalocean.md
index 3af5f97e..b0c209a2 100644
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@@ -63,7 +63,7 @@ provider "ct" {
Fedora CoreOS publishes images for DigitalOcean, but does not yet upload them. DigitalOcean allows [custom images](https://blog.digitalocean.com/custom-images/) to be uploaded via URL or file.
-Import a [Fedora CoreOS](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) image via URL to desired a region(s). Reference the DigitalOcean image and set the `os_image` in the next step.
+Import a [Fedora CoreOS](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) image via URL to desired a region(s).
```tf
data "digitalocean_image" "fedora-coreos-31-20200323-3-2" {
@@ -71,6 +71,8 @@ data "digitalocean_image" "fedora-coreos-31-20200323-3-2" {
}
```
+Set the [os_image](#variables) in the next step.
+
## Cluster
Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubernetes`.
@@ -83,9 +85,9 @@ module "nemo" {
cluster_name = "nemo"
region = "nyc3"
dns_zone = "digital-ocean.example.com"
- os_image = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id
# configuration
+ os_image = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id
ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
# optional
diff --git a/docs/fedora-coreos/google-cloud.md b/docs/fedora-coreos/google-cloud.md
index 7ebcd2e1..482900b3 100644
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@@ -82,6 +82,8 @@ Create a Compute Engine image from the file.
gcloud compute images create fedora-coreos-31-20200323-3-2 --source-uri gs://BUCKET/fedora-coreos-31.20200323.3.2-gcp.x86_64.tar.gz
```
+Set the [os_image](#variables) in the next step.
+
## Cluster
Define a Kubernetes cluster using the module `google-cloud/fedora-coreos/kubernetes`.
@@ -96,10 +98,8 @@ module "yavin" {
dns_zone = "example.com"
dns_zone_name = "example-zone"
- # custom image name from above
- os_image = "fedora-coreos-31-20200323-3-2"
-
# configuration
+ os_image = "fedora-coreos-31-20200323-3-2"
ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
# optional
diff --git a/docs/index.md b/docs/index.md
index f3407483..b27c4017 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -26,6 +26,7 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
| Platform | Operating System | Terraform Module | Status |
|---------------|------------------|------------------|--------|
| AWS | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | stable |
+| Azure | Fedora CoreOS | [azure/fedora-coreos/kubernetes](fedora-coreos/azure.md) | alpha |
| Bare-Metal | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | beta |
| DigitalOcean | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](fedora-coreos/digitalocean.md) | alpha |
| Google Cloud | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |
@@ -54,7 +55,7 @@ Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com
## Documentation
* Architecture [concepts](architecture/concepts.md) and [operating-systems](architecture/operating-systems.md)
-* Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Bare-Metal](fedora-coreos/bare-metal.md), [DigitalOcean](fedora-coreos/digitalocean.md), and [Google Cloud](fedora-coreos/google-cloud.md)
+* Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Azure](fedora-coreos/azure.md), [Bare-Metal](fedora-coreos/bare-metal.md), [DigitalOcean](fedora-coreos/digitalocean.md), and [Google Cloud](fedora-coreos/google-cloud.md)
* Flatcar Linux tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [DigitalOcean](cl/digital-ocean.md), and [Google Cloud](cl/google-cloud.md)
## Example
diff --git a/google-cloud/fedora-coreos/kubernetes/README.md b/google-cloud/fedora-coreos/kubernetes/README.md
index 94cd7d8a..1eb91836 100644
--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@@ -14,10 +14,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
* Kubernetes v1.18.1 (upstream)
* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
## Docs
-Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/cl/google-cloud/).
+Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/fedora-coreos/google-cloud/).