Fix race condition creating DigitalOcean firewall rules

* DigitalOcean firewall rules should reference Terraform tag resources rather than using tag strings. Otherwise, terraform apply can fail (neeeds rerun) if a tag has not yet been created
Fix bootstrap mount to use shared volume SELinux label
2025-08-02 19:01:34 +02:00 · 2020-04-19 16:55:02 -07:00 · 2020-04-19 16:31:32 -07:00 · 2020-04-18 15:40:57 -07:00 · 2020-04-16 23:49:55 -07:00 · 2020-04-16 23:40:52 -07:00
84 changed files with 2199 additions and 332 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,52 @@ Notable changes between versions.

 ## Latest

+## v1.18.2
+
+* Kubernetes [v1.18.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1182)
+* Choose Fedora CoreOS or Flatcar Linux (**action required**)
+  * Use a `fedora-coreos` module for Fedora CoreOS
+  * Use a `container-linux` module for Flatcar Linux
+* Change Container Linux modules' defaults from CoreOS Container Linux to [Flatcar Container Linux](https://typhoon.psdn.io/architecture/operating-systems/) ([#702](https://github.com/poseidon/typhoon/pull/702))
+  * CoreOS Container Linux [won't receive updates](https://coreos.com/os/eol/) after May 2020
+
+### Fedora CoreOS
+
+* Fix bootstrap race condition from SELinux unshared content label ([#708](https://github.com/poseidon/typhoon/pull/708))
+
+#### Azure
+
+* Add support for Fedora CoreOS ([#704](https://github.com/poseidon/typhoon/pull/704))
+
+#### DigitalOcean
+
+* Fix race condition creating firewall allow rules ([#709](https://github.com/poseidon/typhoon/pull/709))
+
+### Flatcar Linux
+
+#### AWS
+
+* Change `os_image` default from `coreos-stable` to `flatcar-stable` ([#702](https://github.com/poseidon/typhoon/pull/702))
+
+#### Azure
+
+* Change `os_image` to be required. Recommend uploading a Flatcar Linux image (**action required**) ([#702](https://github.com/poseidon/typhoon/pull/702))
+* Disable Flatcar Linux Azure Marketplace image [support](https://github.com/poseidon/typhoon/pull/664) (**breaking**, [#707](https://github.com/poseidon/typhoon/pull/707))
+  * Revert to manual uploading until marketplace issue is closed ([#703](https://github.com/poseidon/typhoon/issues/703))
+
+#### Bare-Metal
+
+* Recommend changing [os_channel](https://typhoon.psdn.io/cl/bare-metal/#required) from `coreos-stable` to `flatcar-stable`
+
+#### Google
+
+* Change `os_image` to be required. Recommend uploading a Flatcar Linux image (**action required**) ([#702](https://github.com/poseidon/typhoon/pull/702))
+
+#### DigitalOcean
+
+* Change `os_image` to be required. Recommend uploading a Flatcar Linux image (**action required**) ([#702](https://github.com/poseidon/typhoon/pull/702))
+* Fix race condition creating firewall allow rules ([#709](https://github.com/poseidon/typhoon/pull/709))
+
 ## v1.18.1

 * Kubernetes [v1.18.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1181)
@ -19,7 +65,7 @@ Notable changes between versions.
 * Rename Container Linux `controller_clc_snippets` to `controller_snippets` for consistency ([#688](https://github.com/poseidon/typhoon/pull/688))
 * Rename Container Linux `worker_clc_snippets` to `worker_snippets` for consistency
 * Rename Container Linux `clc_snippets` (bare-metal) to `snippets` for consistency
-* Drop support for [gitRepo](https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo) volumes
+* Drop support for [gitRepo](https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo) volumes ([kubelet#3](https://github.com/poseidon/kubelet/pull/3))

 #### Azure

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@ -26,6 +26,7 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | stable |
+| Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](azure/fedora-coreos/kubernetes) | alpha |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | alpha |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |
@ -54,7 +55,7 @@ Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com

 * [Docs](https://typhoon.psdn.io)
 * Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
-* Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), [DigitalOcean](docs/fedora-coreos/digitalocean.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
+* Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Azure](docs/fedora-coreos/azure.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), [DigitalOcean](docs/fedora-coreos/digitalocean.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
 * Flatcar Linux tutorials for [AWS](docs/cl/aws.md), [Azure](docs/cl/azure.md), [Bare-Metal](docs/cl/bare-metal.md), [DigitalOcean](docs/cl/digital-ocean.md), and [Google Cloud](docs/cl/google-cloud.md)

 ## Usage
@ -63,7 +64,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.2"

  # Google Cloud
  cluster_name  = "yavin"
@ -102,9 +103,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
 ```

 List the pods.
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/container-linux/kubernetes/cl/controller.yaml
+++ b/aws/container-linux/kubernetes/cl/controller.yaml
@ -91,7 +91,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -134,7 +134,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -169,8 +169,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -184,10 +182,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -44,7 +44,7 @@ variable "worker_type" {
 variable "os_image" {
  type        = string
  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
-  default     = "coreos-stable"
+  default     = "flatcar-stable"
 }

 variable "disk_size" {
@ -96,12 +96,6 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (calico or flannel)"
@ -155,6 +149,12 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml
@ -64,7 +64,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -127,7 +127,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -37,7 +37,7 @@ variable "instance_type" {
 variable "os_image" {
  type        = string
  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
-  default     = "coreos-stable"
+  default     = "flatcar-stable"
 }

 variable "disk_size" {
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -79,7 +79,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -116,13 +116,14 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.1
+            quay.io/poseidon/kubelet:v1.18.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -154,8 +155,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -168,10 +167,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -44,7 +44,7 @@ variable "worker_type" {
 variable "os_image" {
  type        = string
  description = "AMI channel for Fedora CoreOS (not yet used)"
-  default     = "coreos-stable"
+  default     = "stable"
 }

 variable "disk_size" {
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -49,7 +49,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -87,7 +87,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/aws/fedora-coreos/kubernetes/workers/variables.tf
+++ b/aws/fedora-coreos/kubernetes/workers/variables.tf
@ -37,7 +37,7 @@ variable "instance_type" {
 variable "os_image" {
  type        = string
  description = "AMI channel for Fedora CoreOS (not yet used)"
-  default     = "coreos-stable"
+  default     = "stable"
 }

 variable "disk_size" {
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/azure/container-linux/kubernetes/bootstrap.tf
+++ b/azure/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/container-linux/kubernetes/cl/controller.yaml
+++ b/azure/container-linux/kubernetes/cl/controller.yaml
@ -90,7 +90,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -167,8 +167,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -182,10 +180,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -53,23 +53,18 @@ resource "azurerm_linux_virtual_machine" "controllers" {
    storage_account_type = "Premium_LRS"
  }

-  source_image_reference {
-    publisher = local.flavor == "flatcar" ? "Kinvolk" : "CoreOS"
-    offer     = local.flavor == "flatcar" ? "flatcar-container-linux" : "CoreOS"
-    sku       = local.channel
-    version   = "latest"
-  }
-
-  # Gross hack just for Flatcar Linux
-  dynamic "plan" {
-    for_each = local.flavor == "flatcar" ? [1] : []
+  // CoreOS Container Linux or Flatcar Container Linux (manual upload)
+  dynamic "source_image_reference" {
+    for_each = local.flavor == "coreos" ? [1] : []

    content {
-      name      = local.channel
-      publisher = "kinvolk"
-      product   = "flatcar-container-linux"
+      publisher = "CoreOS"
+      offer     = "CoreOS"
+      sku       = local.channel
+      version   = "latest"
    }
  }
+  source_image_id = local.flavor == "coreos" ? null : var.os_image

  # network
  network_interface_ids = [
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -48,8 +48,7 @@ variable "worker_type" {

 variable "os_image" {
  type        = string
-  default     = "coreos-stable"
-  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta)"
+  description = "Channel for a Container Linux derivative (/subscriptions/some-flatcar-upload, coreos-stable, coreos-beta, coreos-alpha)"
 }

 variable "disk_size" {
@ -60,7 +59,7 @@ variable "disk_size" {

 variable "worker_priority" {
  type        = string
-  description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
+  description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
  default     = "Regular"
 }

@ -83,12 +82,6 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (flannel or calico)"
@ -136,6 +129,12 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml
@ -63,7 +63,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -125,7 +125,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@ -46,13 +46,13 @@ variable "vm_type" {

 variable "os_image" {
  type        = string
-  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
-  default     = "coreos-stable"
+  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, coreos-stable, coreos-beta, coreos-alpha)"
+  default     = "flatcar-stable"
 }

 variable "priority" {
  type        = string
-  description = "Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
+  description = "Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
  default     = "Regular"
 }

--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -24,23 +24,18 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
    caching              = "ReadWrite"
  }

-  source_image_reference {
-    publisher = local.flavor == "flatcar" ? "Kinvolk" : "CoreOS"
-    offer     = local.flavor == "flatcar" ? "flatcar-container-linux" : "CoreOS"
-    sku       = local.channel
-    version   = "latest"
-  }
-
-  # Gross hack just for Flatcar Linux
-  dynamic "plan" {
-    for_each = local.flavor == "flatcar" ? [1] : []
+  // CoreOS Container Linux or Flatcar Container Linux (manual upload)
+  dynamic "source_image_reference" {
+    for_each = local.flavor == "coreos" ? [1] : []

    content {
-      name      = local.channel
-      publisher = "kinvolk"
-      product   = "flatcar-container-linux"
+      publisher = "CoreOS"
+      offer     = "CoreOS"
+      sku       = local.channel
+      version   = "latest"
    }
  }
+  source_image_id = local.flavor == "coreos" ? null : var.os_image

  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
  admin_username = "core"
--- a/azure/fedora-coreos/kubernetes/LICENSE
+++ b/azure/fedora-coreos/kubernetes/LICENSE
@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Typhoon Authors
+Copyright (c) 2020 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -0,0 +1,23 @@
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
+
+* Kubernetes v1.18.2 (upstream)
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
+* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/fedora-coreos/azure/).
+
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -0,0 +1,25 @@
+# Kubernetes assets (kubeconfig, manifests)
+module "bootstrap" {
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+
+  cluster_name = var.cluster_name
+  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
+  etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
+  asset_dir    = var.asset_dir
+
+  networking = var.networking
+
+  # only effective with Calico networking
+  network_encapsulation = "vxlan"
+  network_mtu           = "1450"
+
+  pod_cidr              = var.pod_cidr
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  enable_reporting      = var.enable_reporting
+  enable_aggregation    = var.enable_aggregation
+
+  # Fedora CoreOS
+  trusted_certs_dir = "/etc/pki/tls/certs"
+}
+
--- a/azure/fedora-coreos/kubernetes/controllers.tf
+++ b/azure/fedora-coreos/kubernetes/controllers.tf
@ -0,0 +1,151 @@
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "azurerm_dns_a_record" "etcds" {
+  count               = var.controller_count
+  resource_group_name = var.dns_zone_group
+
+  # DNS Zone name where record should be created
+  zone_name = var.dns_zone
+
+  # DNS record
+  name = format("%s-etcd%d", var.cluster_name, count.index)
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  records = [azurerm_network_interface.controllers.*.private_ip_address[count.index]]
+}
+
+# Controller availability set to spread controllers
+resource "azurerm_availability_set" "controllers" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                         = "${var.cluster_name}-controllers"
+  location                     = var.region
+  platform_fault_domain_count  = 2
+  platform_update_domain_count = 4
+  managed                      = true
+}
+
+# Controller instances
+resource "azurerm_linux_virtual_machine" "controllers" {
+  count               = var.controller_count
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                = "${var.cluster_name}-controller-${count.index}"
+  location            = var.region
+  availability_set_id = azurerm_availability_set.controllers.id
+
+  size        = var.controller_type
+  custom_data = base64encode(data.ct_config.controller-ignitions.*.rendered[count.index])
+
+  # storage
+  source_image_id = var.os_image
+  os_disk {
+    name                 = "${var.cluster_name}-controller-${count.index}"
+    caching              = "None"
+    disk_size_gb         = var.disk_size
+    storage_account_type = "Premium_LRS"
+  }
+
+  # network
+  network_interface_ids = [
+    azurerm_network_interface.controllers.*.id[count.index]
+  ]
+
+  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
+  admin_username = "core"
+  admin_ssh_key {
+    username   = "core"
+    public_key = var.ssh_authorized_key
+  }
+
+  lifecycle {
+    ignore_changes = [
+      os_disk,
+      custom_data,
+    ]
+  }
+}
+
+# Controller public IPv4 addresses
+resource "azurerm_public_ip" "controllers" {
+  count               = var.controller_count
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name              = "${var.cluster_name}-controller-${count.index}"
+  location          = azurerm_resource_group.cluster.location
+  sku               = "Standard"
+  allocation_method = "Static"
+}
+
+# Controller NICs with public and private IPv4
+resource "azurerm_network_interface" "controllers" {
+  count               = var.controller_count
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name     = "${var.cluster_name}-controller-${count.index}"
+  location = azurerm_resource_group.cluster.location
+
+  ip_configuration {
+    name                          = "ip0"
+    subnet_id                     = azurerm_subnet.controller.id
+    private_ip_address_allocation = "Dynamic"
+    # instance public IPv4
+    public_ip_address_id = azurerm_public_ip.controllers.*.id[count.index]
+  }
+}
+
+# Associate controller network interface with controller security group
+resource "azurerm_network_interface_security_group_association" "controllers" {
+  count = var.controller_count
+
+  network_interface_id      = azurerm_network_interface.controllers[count.index].id
+  network_security_group_id = azurerm_network_security_group.controller.id
+}
+
+# Associate controller network interface with controller backend address pool
+resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+  count = var.controller_count
+
+  network_interface_id    = azurerm_network_interface.controllers[count.index].id
+  ip_configuration_name   = "ip0"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+}
+
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = var.controller_count
+  content      = data.template_file.controller-configs.*.rendered[count.index]
+  pretty_print = false
+  snippets     = var.controller_snippets
+}
+
+# Controller Fedora CoreOS configs
+data "template_file" "controller-configs" {
+  count = var.controller_count
+
+  template = file("${path.module}/fcc/controller.yaml")
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+  }
+}
+
+data "template_file" "etcds" {
+  count    = var.controller_count
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+  vars = {
+    index        = count.index
+    cluster_name = var.cluster_name
+    dns_zone     = var.dns_zone
+  }
+}
+
--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -0,0 +1,213 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/coreos/etcd
+        Wants=network-online.target network.target
+        After=network-online.target
+        [Service]
+        # https://github.com/opencontainers/runc/pull/1807
+        # Type=notify
+        # NotifyAccess=exec
+        Type=exec
+        Restart=on-failure
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        ExecStartPre=/bin/mkdir -p /var/lib/etcd
+        ExecStartPre=-/usr/bin/podman rm etcd
+        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
+        ExecStart=/usr/bin/podman run --name etcd \
+          --env-file /etc/etcd/etcd.env \
+          --network host \
+          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
+          quay.io/coreos/etcd:v3.4.7
+        ExecStop=/usr/bin/podman stop etcd
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube (System Container)
+        Wants=rpc-statd.service
+        [Service]
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/podman rm kubelet
+        ExecStart=/usr/bin/podman run --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /usr/lib/os-release:/etc/os-release:ro \
+          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /lib/modules:/lib/modules:ro \
+          --volume /run:/run \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /var/lib/calico:/var/lib/calico:ro \
+          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+          --volume /var/log:/var/log \
+          --volume /var/run/lock:/var/run/lock:z \
+          --volume /opt/cni/bin:/opt/cni/bin:z \
+          quay.io/poseidon/kubelet:v1.18.2 \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --cgroup-driver=systemd \
+          --cgroups-per-qos=true \
+          --enforce-node-allocatable=pods \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --healthz-port=0 \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/master \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/podman stop kubelet
+        Delegate=yes
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/podman rm bootstrap
+        ExecStart=/usr/bin/podman run --name bootstrap \
+            --network host \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
+            --volume /opt/bootstrap/assets:/assets:ro,Z \
+            --volume /opt/bootstrap/apply:/apply:ro,Z \
+            --entrypoint=/apply \
+            quay.io/poseidon/kubelet:v1.18.2
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        ExecStartPost=-/usr/bin/podman stop bootstrap
+storage:
+  directories:
+    - path: /etc/kubernetes
+    - path: /opt/bootstrap
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /opt/bootstrap/layout
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          sudo mkdir -p /etc/kubernetes/manifests
+          sudo mv static-manifests/* /etc/kubernetes/manifests/
+          sudo mkdir -p /opt/bootstrap/assets
+          sudo mv manifests /opt/bootstrap/assets/manifests
+          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests-networking
+    - path: /opt/bootstrap/apply
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/sysctl.d/max-user-watches.conf
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/systemd/system.conf.d/accounting.conf
+      contents:
+        inline: |
+          [Manager]
+          DefaultCPUAccounting=yes
+          DefaultMemoryAccounting=yes
+          DefaultBlockIOAccounting=yes
+    - path: /etc/etcd/etcd.env
+      mode: 0644
+      contents:
+        inline: |
+          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
+          NOTIFY_SOCKET=/run/systemd/notify
+          ETCD_NAME=${etcd_name}
+          ETCD_DATA_DIR=/var/lib/etcd
+          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+          ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+          ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+          ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+          ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+          ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+          ETCD_STRICT_RECONFIG_CHECK=true
+          ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+          ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+          ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+          ETCD_CLIENT_CERT_AUTH=true
+          ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+          ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - ${ssh_authorized_key}
+
--- a/azure/fedora-coreos/kubernetes/lb.tf
+++ b/azure/fedora-coreos/kubernetes/lb.tf
@ -0,0 +1,161 @@
+# DNS record for the apiserver load balancer
+resource "azurerm_dns_a_record" "apiserver" {
+  resource_group_name = var.dns_zone_group
+
+  # DNS Zone name where record should be created
+  zone_name = var.dns_zone
+
+  # DNS record
+  name = var.cluster_name
+  ttl  = 300
+
+  # IPv4 address of apiserver load balancer
+  records = [azurerm_public_ip.apiserver-ipv4.ip_address]
+}
+
+# Static IPv4 address for the apiserver frontend
+resource "azurerm_public_ip" "apiserver-ipv4" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name              = "${var.cluster_name}-apiserver-ipv4"
+  location          = var.region
+  sku               = "Standard"
+  allocation_method = "Static"
+}
+
+# Static IPv4 address for the ingress frontend
+resource "azurerm_public_ip" "ingress-ipv4" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name              = "${var.cluster_name}-ingress-ipv4"
+  location          = var.region
+  sku               = "Standard"
+  allocation_method = "Static"
+}
+
+# Network Load Balancer for apiservers and ingress
+resource "azurerm_lb" "cluster" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name     = var.cluster_name
+  location = var.region
+  sku      = "Standard"
+
+  frontend_ip_configuration {
+    name                 = "apiserver"
+    public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
+  }
+
+  frontend_ip_configuration {
+    name                 = "ingress"
+    public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
+  }
+}
+
+resource "azurerm_lb_rule" "apiserver" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                           = "apiserver"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "apiserver"
+
+  protocol                = "Tcp"
+  frontend_port           = 6443
+  backend_port            = 6443
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  probe_id                = azurerm_lb_probe.apiserver.id
+}
+
+resource "azurerm_lb_rule" "ingress-http" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                           = "ingress-http"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "ingress"
+  disable_outbound_snat          = true
+
+  protocol                = "Tcp"
+  frontend_port           = 80
+  backend_port            = 80
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  probe_id                = azurerm_lb_probe.ingress.id
+}
+
+resource "azurerm_lb_rule" "ingress-https" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                           = "ingress-https"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "ingress"
+  disable_outbound_snat          = true
+
+  protocol                = "Tcp"
+  frontend_port           = 443
+  backend_port            = 443
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  probe_id                = azurerm_lb_probe.ingress.id
+}
+
+# Worker outbound TCP/UDP SNAT
+resource "azurerm_lb_outbound_rule" "worker-outbound" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name            = "worker"
+  loadbalancer_id = azurerm_lb.cluster.id
+  frontend_ip_configuration {
+    name = "ingress"
+  }
+
+  protocol                = "All"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+}
+
+# Address pool of controllers
+resource "azurerm_lb_backend_address_pool" "controller" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name            = "controller"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+# Address pool of workers
+resource "azurerm_lb_backend_address_pool" "worker" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name            = "worker"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+# Health checks / probes
+
+# TCP health check for apiserver
+resource "azurerm_lb_probe" "apiserver" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name            = "apiserver"
+  loadbalancer_id = azurerm_lb.cluster.id
+  protocol        = "Tcp"
+  port            = 6443
+
+  # unhealthy threshold
+  number_of_probes = 3
+
+  interval_in_seconds = 5
+}
+
+# HTTP health check for ingress
+resource "azurerm_lb_probe" "ingress" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name            = "ingress"
+  loadbalancer_id = azurerm_lb.cluster.id
+  protocol        = "Http"
+  port            = 10254
+  request_path    = "/healthz"
+
+  # unhealthy threshold
+  number_of_probes = 3
+
+  interval_in_seconds = 5
+}
+
--- a/azure/fedora-coreos/kubernetes/network.tf
+++ b/azure/fedora-coreos/kubernetes/network.tf
@ -0,0 +1,44 @@
+# Organize cluster into a resource group
+resource "azurerm_resource_group" "cluster" {
+  name     = var.cluster_name
+  location = var.region
+}
+
+resource "azurerm_virtual_network" "network" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name          = var.cluster_name
+  location      = azurerm_resource_group.cluster.location
+  address_space = [var.host_cidr]
+}
+
+# Subnets - separate subnets for controller and workers because Azure
+# network security groups are based on IPv4 CIDR rather than instance
+# tags like GCP or security group membership like AWS
+
+resource "azurerm_subnet" "controller" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                 = "controller"
+  virtual_network_name = azurerm_virtual_network.network.name
+  address_prefix       = cidrsubnet(var.host_cidr, 1, 0)
+}
+
+resource "azurerm_subnet_network_security_group_association" "controller" {
+  subnet_id                 = azurerm_subnet.controller.id
+  network_security_group_id = azurerm_network_security_group.controller.id
+}
+
+resource "azurerm_subnet" "worker" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                 = "worker"
+  virtual_network_name = azurerm_virtual_network.network.name
+  address_prefix       = cidrsubnet(var.host_cidr, 1, 1)
+}
+
+resource "azurerm_subnet_network_security_group_association" "worker" {
+  subnet_id                 = azurerm_subnet.worker.id
+  network_security_group_id = azurerm_network_security_group.worker.id
+}
+
--- a/azure/fedora-coreos/kubernetes/outputs.tf
+++ b/azure/fedora-coreos/kubernetes/outputs.tf
@ -0,0 +1,59 @@
+output "kubeconfig-admin" {
+  value = module.bootstrap.kubeconfig-admin
+}
+
+# Outputs for Kubernetes Ingress
+
+output "ingress_static_ipv4" {
+  value       = azurerm_public_ip.ingress-ipv4.ip_address
+  description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
+# Outputs for worker pools
+
+output "region" {
+  value = azurerm_resource_group.cluster.location
+}
+
+output "resource_group_name" {
+  value = azurerm_resource_group.cluster.name
+}
+
+output "resource_group_id" {
+  value = azurerm_resource_group.cluster.id
+}
+
+output "subnet_id" {
+  value = azurerm_subnet.worker.id
+}
+
+output "security_group_id" {
+  value = azurerm_network_security_group.worker.id
+}
+
+output "kubeconfig" {
+  value = module.bootstrap.kubeconfig-kubelet
+}
+
+# Outputs for custom firewalling
+
+output "worker_security_group_name" {
+  value = azurerm_network_security_group.worker.name
+}
+
+output "worker_address_prefix" {
+  description = "Worker network subnet CIDR address (for source/destination)"
+  value       = azurerm_subnet.worker.address_prefix
+}
+
+# Outputs for custom load balancing
+
+output "loadbalancer_id" {
+  description = "ID of the cluster load balancer"
+  value       = azurerm_lb.cluster.id
+}
+
+output "backend_address_pool_id" {
+  description = "ID of the worker backend address pool"
+  value       = azurerm_lb_backend_address_pool.worker.id
+}
--- a/azure/fedora-coreos/kubernetes/security.tf
+++ b/azure/fedora-coreos/kubernetes/security.tf
@ -0,0 +1,336 @@
+# Controller security group
+
+resource "azurerm_network_security_group" "controller" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name     = "${var.cluster_name}-controller"
+  location = azurerm_resource_group.cluster.location
+}
+
+resource "azurerm_network_security_rule" "controller-ssh" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-ssh"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "22"
+  source_address_prefix       = "*"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+resource "azurerm_network_security_rule" "controller-etcd" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-etcd"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2005"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "2379-2380"
+  source_address_prefix       = azurerm_subnet.controller.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape etcd metrics
+resource "azurerm_network_security_rule" "controller-etcd-metrics" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-etcd-metrics"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2010"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "2381"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape kube-proxy metrics
+resource "azurerm_network_security_rule" "controller-kube-proxy" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kube-proxy-metrics"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2011"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10249"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
+resource "azurerm_network_security_rule" "controller-kube-metrics" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kube-metrics"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2012"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10251-10252"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+resource "azurerm_network_security_rule" "controller-apiserver" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-apiserver"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2015"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "6443"
+  source_address_prefix       = "*"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+resource "azurerm_network_security_rule" "controller-vxlan" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-vxlan"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2020"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Udp"
+  source_port_range           = "*"
+  destination_port_range      = "4789"
+  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "controller-node-exporter" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-node-exporter"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2025"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "9100"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "controller-kubelet" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kubelet"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2030"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10250"
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix = azurerm_subnet.controller.address_prefix
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-loadbalancer"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "3000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "AzureLoadBalancer"
+  destination_address_prefix  = "*"
+}
+
+resource "azurerm_network_security_rule" "controller-deny-all" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "deny-all"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "3005"
+  access                      = "Deny"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "*"
+}
+
+# Worker security group
+
+resource "azurerm_network_security_group" "worker" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name     = "${var.cluster_name}-worker"
+  location = azurerm_resource_group.cluster.location
+}
+
+resource "azurerm_network_security_rule" "worker-ssh" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-ssh"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "22"
+  source_address_prefix       = azurerm_subnet.controller.address_prefix
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
+resource "azurerm_network_security_rule" "worker-http" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-http"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2005"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "80"
+  source_address_prefix       = "*"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
+resource "azurerm_network_security_rule" "worker-https" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-https"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2010"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "443"
+  source_address_prefix       = "*"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
+resource "azurerm_network_security_rule" "worker-vxlan" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-vxlan"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2015"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Udp"
+  source_port_range           = "*"
+  destination_port_range      = "4789"
+  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "worker-node-exporter" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-node-exporter"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2020"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "9100"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
+# Allow Prometheus to scrape kube-proxy
+resource "azurerm_network_security_rule" "worker-kube-proxy" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kube-proxy"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2024"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10249"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "worker-kubelet" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kubelet"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2025"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10250"
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix = azurerm_subnet.worker.address_prefix
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-loadbalancer"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "3000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "AzureLoadBalancer"
+  destination_address_prefix  = "*"
+}
+
+resource "azurerm_network_security_rule" "worker-deny-all" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "deny-all"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "3005"
+  access                      = "Deny"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "*"
+}
+
--- a/azure/fedora-coreos/kubernetes/ssh.tf
+++ b/azure/fedora-coreos/kubernetes/ssh.tf
@ -0,0 +1,59 @@
+locals {
+  # format assets for distribution
+  assets_bundle = [
+    # header with the unpack location
+    for key, value in module.bootstrap.assets_dist :
+    format("##### %s\n%s", key, value)
+  ]
+}
+
+# Secure copy assets to controllers.
+resource "null_resource" "copy-controller-secrets" {
+  count = var.controller_count
+
+  depends_on = [
+    module.bootstrap,
+    azurerm_linux_virtual_machine.controllers
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = azurerm_public_ip.controllers.*.ip_address[count.index]
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = join("\n", local.assets_bundle)
+    destination = "$HOME/assets"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo /opt/bootstrap/layout",
+    ]
+  }
+}
+
+# Connect to a controller to perform one-time cluster bootstrap.
+resource "null_resource" "bootstrap" {
+  depends_on = [
+    null_resource.copy-controller-secrets,
+    module.workers,
+    azurerm_dns_a_record.apiserver,
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = azurerm_public_ip.controllers.*.ip_address[0]
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo systemctl start bootstrap",
+    ]
+  }
+}
+
--- a/azure/fedora-coreos/kubernetes/variables.tf
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@ -0,0 +1,143 @@
+variable "cluster_name" {
+  type        = string
+  description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# Azure
+
+variable "region" {
+  type        = string
+  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+}
+
+variable "dns_zone" {
+  type        = string
+  description = "Azure DNS Zone (e.g. azure.example.com)"
+}
+
+variable "dns_zone_group" {
+  type        = string
+  description = "Resource group where the Azure DNS Zone resides (e.g. global)"
+}
+
+# instances
+
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
+  default     = "Standard_B2s"
+}
+
+variable "worker_type" {
+  type        = string
+  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
+  default     = "Standard_DS1_v2"
+}
+
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for instances"
+}
+
+variable "disk_size" {
+  type        = number
+  description = "Size of the disk in GB"
+  default     = 40
+}
+
+variable "worker_priority" {
+  type        = string
+  description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
+  default     = "Regular"
+}
+
+variable "controller_snippets" {
+  type        = list(string)
+  description = "Controller Fedora CoreOS Config snippets"
+  default     = []
+}
+
+variable "worker_snippets" {
+  type        = list(string)
+  description = "Worker Fedora CoreOS Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+  type        = string
+  description = "SSH public key for user 'core'"
+}
+
+variable "networking" {
+  type        = string
+  description = "Choice of networking provider (flannel or calico)"
+  default     = "calico"
+}
+
+variable "host_cidr" {
+  type        = string
+  description = "CIDR IPv4 range to assign to instances"
+  default     = "10.0.0.0/16"
+}
+
+variable "pod_cidr" {
+  type        = string
+  description = "CIDR IPv4 range to assign Kubernetes pods"
+  default     = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+  type        = string
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+  default     = "10.3.0.0/16"
+}
+
+variable "enable_reporting" {
+  type        = bool
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = false
+}
+
+variable "enable_aggregation" {
+  type        = bool
+  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  default     = false
+}
+
+variable "worker_node_labels" {
+  type        = list(string)
+  description = "List of initial worker node labels"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
+variable "cluster_domain_suffix" {
+  type        = string
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  default     = "cluster.local"
+}
+
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.6"
+  required_providers {
+    azurerm  = "~> 2.0"
+    ct       = "~> 0.3"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}
+
--- a/azure/fedora-coreos/kubernetes/workers.tf
+++ b/azure/fedora-coreos/kubernetes/workers.tf
@ -0,0 +1,24 @@
+module "workers" {
+  source = "./workers"
+  name   = var.cluster_name
+
+  # Azure
+  resource_group_name     = azurerm_resource_group.cluster.name
+  region                  = azurerm_resource_group.cluster.location
+  subnet_id               = azurerm_subnet.worker.id
+  security_group_id       = azurerm_network_security_group.worker.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+
+  worker_count = var.worker_count
+  vm_type      = var.worker_type
+  os_image     = var.os_image
+  priority     = var.worker_priority
+
+  # configuration
+  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  ssh_authorized_key    = var.ssh_authorized_key
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets              = var.worker_snippets
+  node_labels           = var.worker_node_labels
+}
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -0,0 +1,119 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+  units:
+    - name: docker.service
+      enabled: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube (System Container)
+        Wants=rpc-statd.service
+        [Service]
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/podman rm kubelet
+        ExecStart=/usr/bin/podman run --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /usr/lib/os-release:/etc/os-release:ro \
+          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /lib/modules:/lib/modules:ro \
+          --volume /run:/run \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /var/lib/calico:/var/lib/calico:ro \
+          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+          --volume /var/log:/var/log \
+          --volume /var/run/lock:/var/run/lock:z \
+          --volume /opt/cni/bin:/opt/cni/bin:z \
+          quay.io/poseidon/kubelet:v1.18.2 \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --cgroup-driver=systemd \
+          --cgroups-per-qos=true \
+          --enforce-node-allocatable=pods \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --healthz-port=0 \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/node \
+          %{~ for label in split(",", node_labels) ~}
+          --node-labels=${label} \
+          %{~ endfor ~}
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/podman stop kubelet
+        Delegate=yes
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: delete-node.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Delete Kubernetes node on shutdown
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/true
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /etc/kubernetes
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/sysctl.d/max-user-watches.conf
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/systemd/system.conf.d/accounting.conf
+      contents:
+        inline: |
+          [Manager]
+          DefaultCPUAccounting=yes
+          DefaultMemoryAccounting=yes
+          DefaultBlockIOAccounting=yes
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - ${ssh_authorized_key}
+
+
--- a/azure/fedora-coreos/kubernetes/workers/variables.tf
+++ b/azure/fedora-coreos/kubernetes/workers/variables.tf
@ -0,0 +1,98 @@
+variable "name" {
+  type        = string
+  description = "Unique name for the worker pool"
+}
+
+# Azure
+
+variable "region" {
+  type        = string
+  description = "Must be set to the Azure Region of cluster"
+}
+
+variable "resource_group_name" {
+  type        = string
+  description = "Must be set to the resource group name of cluster"
+}
+
+variable "subnet_id" {
+  type        = string
+  description = "Must be set to the `worker_subnet_id` output by cluster"
+}
+
+variable "security_group_id" {
+  type        = string
+  description = "Must be set to the `worker_security_group_id` output by cluster"
+}
+
+variable "backend_address_pool_id" {
+  type        = string
+  description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
+}
+
+# instances
+
+variable "worker_count" {
+  type        = number
+  description = "Number of instances"
+  default     = 1
+}
+
+variable "vm_type" {
+  type        = string
+  description = "Machine type for instances (see `az vm list-skus --location centralus`)"
+  default     = "Standard_DS1_v2"
+}
+
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for instances"
+}
+
+variable "priority" {
+  type        = string
+  description = "Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
+  default     = "Regular"
+}
+
+variable "snippets" {
+  type        = list(string)
+  description = "Fedora CoreOS Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "kubeconfig" {
+  type        = string
+  description = "Must be set to `kubeconfig` output by cluster"
+}
+
+variable "ssh_authorized_key" {
+  type        = string
+  description = "SSH public key for user 'core'"
+}
+
+variable "service_cidr" {
+  type        = string
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+  default     = "10.3.0.0/16"
+}
+
+variable "node_labels" {
+  type        = list(string)
+  description = "List of initial node labels"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = string
+  default     = "cluster.local"
+}
+
--- a/azure/fedora-coreos/kubernetes/workers/versions.tf
+++ b/azure/fedora-coreos/kubernetes/workers/versions.tf
@ -0,0 +1,4 @@
+
+terraform {
+  required_version = ">= 0.12"
+}
--- a/azure/fedora-coreos/kubernetes/workers/workers.tf
+++ b/azure/fedora-coreos/kubernetes/workers/workers.tf
@ -0,0 +1,92 @@
+# Workers scale set
+resource "azurerm_linux_virtual_machine_scale_set" "workers" {
+  resource_group_name = var.resource_group_name
+
+  name      = "${var.name}-worker"
+  location  = var.region
+  sku       = var.vm_type
+  instances = var.worker_count
+  # instance name prefix for instances in the set
+  computer_name_prefix   = "${var.name}-worker"
+  single_placement_group = false
+  custom_data            = base64encode(data.ct_config.worker-ignition.rendered)
+
+  # storage
+  source_image_id = var.os_image
+  os_disk {
+    storage_account_type = "Standard_LRS"
+    caching              = "ReadWrite"
+  }
+
+  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
+  admin_username = "core"
+  admin_ssh_key {
+    username   = "core"
+    public_key = var.ssh_authorized_key
+  }
+
+  # network
+  network_interface {
+    name                      = "nic0"
+    primary                   = true
+    network_security_group_id = var.security_group_id
+
+    ip_configuration {
+      name      = "ip0"
+      primary   = true
+      subnet_id = var.subnet_id
+
+      # backend address pool to which the NIC should be added
+      load_balancer_backend_address_pool_ids = [var.backend_address_pool_id]
+    }
+  }
+
+  # lifecycle
+  upgrade_mode = "Manual"
+  # eviction policy may only be set when priority is Spot
+  priority        = var.priority
+  eviction_policy = var.priority == "Spot" ? "Delete" : null
+}
+
+# Scale up or down to maintain desired number, tolerating deallocations.
+resource "azurerm_monitor_autoscale_setting" "workers" {
+  resource_group_name = var.resource_group_name
+
+  name     = "${var.name}-maintain-desired"
+  location = var.region
+
+  # autoscale
+  enabled            = true
+  target_resource_id = azurerm_linux_virtual_machine_scale_set.workers.id
+
+  profile {
+    name = "default"
+
+    capacity {
+      minimum = var.worker_count
+      default = var.worker_count
+      maximum = var.worker_count
+    }
+  }
+}
+
+# Worker Ignition configs
+data "ct_config" "worker-ignition" {
+  content      = data.template_file.worker-config.rendered
+  pretty_print = false
+  snippets     = var.snippets
+}
+
+# Worker Fedora CoreOS configs
+data "template_file" "worker-config" {
+  template = file("${path.module}/fcc/worker.yaml")
+
+  vars = {
+    kubeconfig             = indent(10, var.kubeconfig)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    node_labels            = join(",", var.node_labels)
+  }
+}
+
--- a/azure/ignore/.gitkeep
+++ b/azure/ignore/.gitkeep
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/container-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml
@ -103,7 +103,7 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -147,7 +147,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -185,8 +185,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -200,10 +198,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml
@ -76,7 +76,7 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
--- a/bare-metal/container-linux/kubernetes/variables.tf
+++ b/bare-metal/container-linux/kubernetes/variables.tf
@ -79,12 +79,6 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (flannel or calico)"
@ -158,6 +152,12 @@ variable "enable_aggregation" {

 # unofficial, undocumented, unsupported

+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -80,7 +80,7 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -127,13 +127,14 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.1
+            quay.io/poseidon/kubelet:v1.18.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -165,8 +166,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -179,10 +178,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -50,7 +50,7 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/digital-ocean/container-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml
@ -101,7 +101,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -144,7 +144,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -176,8 +176,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -191,10 +189,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml
@ -74,7 +74,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -131,7 +131,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/network.tf
+++ b/digital-ocean/container-linux/kubernetes/network.tf
@ -1,7 +1,10 @@
 resource "digitalocean_firewall" "rules" {
  name = var.cluster_name

-  tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  tags = [
+    digitalocean_tag.controllers.name,
+    digitalocean_tag.workers.name
+  ]

  # allow ssh, internal flannel, internal node-exporter, internal kubelet
  inbound_rule {
@ -59,7 +62,7 @@ resource "digitalocean_firewall" "rules" {
 resource "digitalocean_firewall" "controllers" {
  name = "${var.cluster_name}-controllers"

-  tags = ["${var.cluster_name}-controller"]
+  tags = [digitalocean_tag.controllers.name]

  # etcd
  inbound_rule {
@ -93,7 +96,7 @@ resource "digitalocean_firewall" "controllers" {
 resource "digitalocean_firewall" "workers" {
  name = "${var.cluster_name}-workers"

-  tags = ["${var.cluster_name}-worker"]
+  tags = [digitalocean_tag.workers.name]

  # allow HTTP/HTTPS ingress
  inbound_rule {
--- a/digital-ocean/container-linux/kubernetes/variables.tf
+++ b/digital-ocean/container-linux/kubernetes/variables.tf
@ -43,8 +43,7 @@ variable "worker_type" {

 variable "os_image" {
  type        = string
-  description = "Container Linux image for instances (e.g. coreos-stable)"
-  default     = "coreos-stable"
+  description = "Container Linux image for instances (e.g. coreos-stable, custom-image-id)"
 }

 variable "controller_snippets" {
@ -66,12 +65,6 @@ variable "ssh_fingerprints" {
  description = "SSH public key fingerprints. (e.g. see `ssh-add -l -E md5`)"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (flannel or calico)"
@ -107,6 +100,12 @@ variable "enable_aggregation" {

 # unofficial, undocumented, unsupported

+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/) customization
--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -81,7 +81,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -131,11 +131,11 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.1
+            quay.io/poseidon/kubelet:v1.18.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -52,7 +52,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -97,7 +97,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/digital-ocean/fedora-coreos/kubernetes/network.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/network.tf
@ -1,7 +1,10 @@
 resource "digitalocean_firewall" "rules" {
  name = var.cluster_name

-  tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  tags = [
+    digitalocean_tag.controllers.name,
+    digitalocean_tag.workers.name
+  ]

  # allow ssh, internal flannel, internal node-exporter, internal kubelet
  inbound_rule {
@ -59,7 +62,7 @@ resource "digitalocean_firewall" "rules" {
 resource "digitalocean_firewall" "controllers" {
  name = "${var.cluster_name}-controllers"

-  tags = ["${var.cluster_name}-controller"]
+  tags = [digitalocean_tag.controllers.name]

  # etcd
  inbound_rule {
@ -93,7 +96,7 @@ resource "digitalocean_firewall" "controllers" {
 resource "digitalocean_firewall" "workers" {
  name = "${var.cluster_name}-workers"

-  tags = ["${var.cluster_name}-worker"]
+  tags = [digitalocean_tag.workers.name]

  # allow HTTP/HTTPS ingress
  inbound_rule {
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -5,8 +5,11 @@ Typhoon AWS, Azure, and Google Cloud allow additional groups of workers to be de
 Internal Terraform Modules:

 * `aws/container-linux/kubernetes/workers`
+* `aws/fedora-coreos/kubernetes/workers`
 * `azure/container-linux/kubernetes/workers`
+* `azure/fedora-coreos/kubernetes/workers`
 * `google-cloud/container-linux/kubernetes/workers`
+* `google-cloud/fedora-coreos/kubernetes/workers`

 ## AWS

@ -20,7 +23,7 @@ module "tempest-worker-pool" {
  vpc_id          = module.tempest.vpc_id
  subnet_ids      = module.tempest.subnet_ids
  security_groups = module.tempest.worker_security_groups
-  
+
  # configuration
  name               = "tempest-pool"
  kubeconfig         = module.tempest.kubeconfig
@ -29,7 +32,7 @@ module "tempest-worker-pool" {
  # optional
  worker_count  = 2
  instance_type = "m5.large"
-  os_image      = "coreos-beta"
+  os_image      = "flatcar-beta"
 }
 ```

@ -62,12 +65,12 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | instance_type | EC2 instance type | "t3.small" | "t3.medium" |
-| os_image | AMI channel for a Container Linux derivative | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
+| os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alph, coreos-stable, coreos-beta, coreos-alpha |
 | disk_size | Size of the EBS volume in GB | 40 | 100 |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | spot_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0 | 0.10 |
-| snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |

@ -79,8 +82,8 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define

 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.18.1"
-  
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.18.2"
+
  # Azure
  region                  = module.ramius.region
  resource_group_name     = module.ramius.resource_group_name
@ -89,14 +92,14 @@ module "ramius-worker-pool" {
  backend_address_pool_id = module.ramius.backend_address_pool_id

  # configuration
-  name               = "ramius-low-priority"
+  name               = "ramius-spot"
  kubeconfig         = module.ramius.kubeconfig
  ssh_authorized_key = var.ssh_authorized_key

  # optional
  worker_count = 2
  vm_type      = "Standard_F4"
-  priority     = "Low"
+  priority     = "Spot"
 }
 ```

@ -131,9 +134,9 @@ The Azure internal `workers` module supports a number of [variables](https://git
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | vm_type | Machine type for instances | "Standard_DS1_v2" | See below |
-| os_image | Channel for a Container Linux derivative | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha |
-| priority | Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | "Regular" | "Low" |
-| snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, coreos-stable, coreos-beta, coreos-alpha |
+| priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | "Regular" | "Spot" |
+| snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |

@ -145,7 +148,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.18.2"

  # Google Cloud
  region       = "europe-west2"
@ -156,7 +159,7 @@ module "yavin-worker-pool" {
  name               = "yavin-16x"
  kubeconfig         = module.yavin.kubeconfig
  ssh_authorized_key = var.ssh_authorized_key
-  
+
  # optional
  worker_count = 2
  machine_type = "n1-standard-16"
@ -176,11 +179,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.18.1
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.18.1
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.18.1
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.18.1
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.18.1
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.18.2
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.18.2
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.18.2
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.18.2
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.18.2
 ```

 ### Variables
@ -196,6 +199,7 @@ The Google Cloud internal `workers` module supports a number of [variables](http
 | region | Region for the worker pool instances. May differ from the cluster's region | "europe-west2" |
 | network | Must be set to `network_name` output by cluster | module.cluster.network_name |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
+| os_image | Container Linux image for compute instances | "fedora-coreos-or-flatcar-image", coreos-stable, coreos-beta, coreos-alpha |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

 Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`.
@ -206,10 +210,9 @@ Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-z
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | machine_type | Compute instance machine type | "n1-standard-1" | See below |
-| os_image | Container Linux image for compute instances | "coreos-stable" | "coreos-alpha", "coreos-beta" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | preemptible | If true, Compute Engine will terminate instances randomly within 24 hours | false | true |
-| snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |

--- a/docs/cl/aws.md
+++ b/docs/cl/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on AWS with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on AWS with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.18.2"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.18.1
-ip-10-0-26-65  Ready   <none>  10m  v1.18.1
-ip-10-0-41-21  Ready   <none>  10m  v1.18.1
+ip-10-0-3-155  Ready   <none>  10m  v1.18.2
+ip-10-0-26-65  Ready   <none>  10m  v1.18.2
+ip-10-0-41-21  Ready   <none>  10m  v1.18.2
 ```

 List the pods.
@ -208,7 +208,7 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
-| os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge |
+| os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge |
 | disk_size | Size of the EBS volume in GB | 40 | 100 |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
--- a/docs/cl/azure.md
+++ b/docs/cl/azure.md
@ -1,9 +1,6 @@
 # Azure

-!!! danger
-    Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.
-
-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on Azure with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Azure with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -50,7 +47,7 @@ Configure the Azure provider in a `providers.tf` file.

 ```tf
 provider "azurerm" {
-  version = "2.1.0"
+  version = "2.5.0"
 }

 provider "ct" {
@ -60,13 +57,45 @@ provider "ct" {

 Additional configuration options are described in the `azurerm` provider [docs](https://www.terraform.io/docs/providers/azurerm/).

+### Flatcar Linux Images
+
+Flatcar Linux publishes images for Azure. Azure allows custom images to be uploaded to a storage account bucket and imported.
+
+[Download](https://www.flatcar-linux.org/releases/) a Flatcar Linux Azure VHD image and upload it to an Azure storage account container (i.e. bucket).
+
+Azure requires fixed VHDs and Flatcar Linux provides dynamic VHDs, so uploads require Azure tools and cannot be done through the UI. Azure's tool compilation requires old versions, so Flatcar Linux has packaged a container image you may choose to use. See their [docs](https://docs.flatcar-linux.org/os/booting-on-azure/#uploading-your-own-image).
+
+```
+bzip2 -d flatcar_production_azure_image.vhd.bz2
+```
+
+```
+podman run -it --entrypoint=/bin/bash quay.io/kinvolk/azure-flatcar-image-upload
+...
+
+# az login
+# az storage account keys list --resource-group GROUP	--account-name BUCKET | jq -r '.[0].value'
+# azure-vhd-utils upload --localvhdpath /data/flatcar_production_azure_image.vhd --stgaccountname BUCKET --containername flatcar-linux --blobname flatcar-stable-2345.3.1 --stgaccountkey "KEYFROMABOVE"
+# exit
+```
+
+Create an Azure disk (note disk ID) and create an Azure image from it (note image ID).
+
+```
+az disk create --name flatcar-stable-2345.3.1 -g GROUP --source https://BUCKET.blob.core.windows.net/flatcar-linux/flatcar_production_azure_image.vhd
+
+az image create --name flatcar-stable-2345.3.1 -g GROUP --os-type=linux --source /subscriptions/some/path/providers/Microsoft.Compute/disks/flatcar-stable-2345.3.1
+```
+
+Set the [os_image](#variables) in the next step.
+
 ## Cluster

 Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.18.2"

  # Azure
  cluster_name   = "ramius"
@ -75,6 +104,7 @@ module "ramius" {
  dns_zone_group = "example-group"

  # configuration
+  os_image           = "/subscriptions/some/path/Microsoft.Compute/images/flatcar-stable-2345.3.1"
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."

  # optional
@ -149,9 +179,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.18.1
-ramius-worker-000001  Ready   <none>  25m  v1.18.1
-ramius-worker-000002  Ready   <none>  24m  v1.18.1
+ramius-controller-0   Ready   <none>  24m  v1.18.2
+ramius-worker-000001  Ready   <none>  25m  v1.18.2
+ramius-worker-000002  Ready   <none>  24m  v1.18.2
 ```

 List the pods.
@ -188,6 +218,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/c
 | region | Azure region | "centralus" |
 | dns_zone | Azure DNS zone | "azure.example.com" |
 | dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
+| os_image | Container Linux image for instances | "/subscriptions/..../some-flatcar-image", coreos-stable, coreos-beta, coreos-alpha |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

 !!! tip
@ -228,7 +259,6 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "Standard_B2s" | See below |
 | worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
-| os_image | Channel for a Container Linux derivative | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
--- a/docs/cl/bare-metal.md
+++ b/docs/cl/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.18.1 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.18.2 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,13 +160,13 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.2"

  # bare-metal
  cluster_name            = "mercury"
  matchbox_http_endpoint  = "http://matchbox.example.com"
-  os_channel              = "coreos-stable"
-  os_version              = "2191.5.0"
+  os_channel              = "flatcar-stable"
+  os_version              = "2345.3.1"

  # configuration
  k8s_domain_name    = "node1.example.com"
@ -299,9 +299,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.18.1
-node2.example.com   Ready   <none>  10m  v1.18.1
-node3.example.com   Ready   <none>  10m  v1.18.1
+node1.example.com   Ready   <none>  10m  v1.18.2
+node2.example.com   Ready   <none>  10m  v1.18.2
+node3.example.com   Ready   <none>  10m  v1.18.2
 ```

 List the pods.
@ -337,7 +337,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | cluster_name | Unique cluster name | "mercury" |
 | matchbox_http_endpoint | Matchbox HTTP read-only endpoint | "http://matchbox.example.com:port" |
 | os_channel | Channel for a Container Linux derivative | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge |
-| os_version | Version for a Container Linux derivative to PXE and install | "1632.3.0" |
+| os_version | Version for a Container Linux derivative to PXE and install | "2345.3.1" |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3Nz..." |
 | controllers | List of controller machine detail objects (unique name, identifying MAC address, FQDN) | `[{name="node1", mac="52:54:00:a1:9c:ae", domain="node1.example.com"}]` |
--- a/docs/cl/digital-ocean.md
+++ b/docs/cl/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -59,21 +59,35 @@ provider "ct" {
 }
 ```

+### Flatcar Linux Images
+
+Flatcar Linux publishes DigitalOcean images, but does not yet upload them. DigitalOcean allows [custom images](https://blog.digitalocean.com/custom-images/) to be uploaded via URLor file.
+
+[Download](https://www.flatcar-linux.org/releases/) the Flatcar Linux DigitalOcean bin image. Rename the image with the channel and version (to refer to these images over time) and [upload](https://cloud.digitalocean.com/images/custom_images) it as a custom image.
+
+```tf
+data "digitalocean_image" "flatcar-stable-2303-4-0" {
+  name = "flatcar-stable-2303.4.0.bin.bz2"
+}
+```
+
+Set the [os_image](#variables) in the next step.
+
 ## Cluster

 Define a Kubernetes cluster using the module `digital-ocean/container-linux/kubernetes`.

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.18.2"

  # Digital Ocean
  cluster_name = "nemo"
  region       = "nyc3"
  dns_zone     = "digital-ocean.example.com"
-  os_image     = "coreos-stable"

  # configuration
+  os_image         = data.digitalocean_image.flatcar-stable-2303-4-0.id
  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]

  # optional
@ -83,28 +97,6 @@ module "nemo" {

 Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/digital-ocean/container-linux/kubernetes/variables.tf) source.

-### Flatcar Linux Only
-
-!!! warning
-    Typhoon for Flatcar Linux on DigitalOcean is alpha. Also IPv6 is unsupported with DigitalOcean custom images.
-
-Flatcar Linux publishes DigitalOcean images, but does not upload them. DigitalOcean allows [custom boot images](https://blog.digitalocean.com/custom-images/) by file or URL.
-
-[Download](https://www.flatcar-linux.org/releases/) the Flatcar Linux DigitalOcean bin image (or copy the URL) and [upload](https://cloud.digitalocean.com/images/custom_images) it as a custom image. Rename the image with the channel and version to refer to these images over time.
-
-```tf
-module "nemo" {
-  ...
-  os_image = data.digitalocean_image.flatcar-stable.id
-}
-
-data "digitalocean_image" "flatcar-stable" {
-  name = "flatcar-stable-2303.4.0.bin.bz2"
-}
-```
-
-Set the [os_image](#variables) to the custom image id.
-
 ## ssh-agent

 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
@ -161,9 +153,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.18.1
-10.132.115.81      Ready   <none>  10m  v1.18.1
-10.132.124.107     Ready   <none>  10m  v1.18.1
+10.132.110.130     Ready   <none>  10m  v1.18.2
+10.132.115.81      Ready   <none>  10m  v1.18.2
+10.132.124.107     Ready   <none>  10m  v1.18.2
 ```

 List the pods.
@ -198,6 +190,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/digital
 | cluster_name | Unique cluster name (prepended to dns_zone) | "nemo" |
 | region | Digital Ocean region | "nyc1", "sfo2", "fra1", tor1" |
 | dns_zone | Digital Ocean domain (i.e. DNS zone) | "do.example.com" |
+| os_image | Container Linux image for instances | "custom-image-id", coreos-stable, coreos-beta, coreos-alpha |
 | ssh_fingerprints | SSH public key fingerprints | ["d7:9d..."] |

 #### DNS Zone
@ -243,7 +236,6 @@ Digital Ocean requires the SSH public key be uploaded to your account, so you ma
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Droplet type for controllers | "s-2vcpu-2gb" | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
 | worker_type | Droplet type for workers | "s-1vcpu-2gb" | s-1vcpu-2gb, s-2vcpu-2gb, ... |
-| os_image | Container Linux image for instances | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha, "custom-image-id" |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "flannel" or "calico" |
--- a/docs/cl/google-cloud.md
+++ b/docs/cl/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -65,36 +65,9 @@ Additional configuration options are described in the `google` provider [docs](h
 !!! tip
    Regions are listed in [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`. A project may contain multiple clusters across different regions.

-## Cluster
+### Flatcar Linux Images

-Define a Kubernetes cluster using the module `google-cloud/container-linux/kubernetes`.
-
-```tf
-module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.1"
-
-  # Google Cloud
-  cluster_name  = "yavin"
-  region        = "us-central1"
-  dns_zone      = "example.com"
-  dns_zone_name = "example-zone"
-
-  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-
-  # optional
-  worker_count = 2
-}
-```
-
-Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.
-
-### Flatcar Linux Only
-
-!!! warning
-    Typhoon for Flatcar Linux on Google Cloud is alpha.
-
-Flatcar Linux publishes Google Cloud images, but does not upload them. Google Cloud allows [custom boot images](https://cloud.google.com/compute/docs/images/import-existing-image) to be uploaded to a bucket and imported into a project.
+Flatcar Linux publishes Google Cloud images, but does not yet upload them. Google Cloud allows [custom boot images](https://cloud.google.com/compute/docs/images/import-existing-image) to be uploaded to a bucket and imported into your project.

 [Download](https://www.flatcar-linux.org/releases/) the Flatcar Linux GCE gzipped tarball and upload it to a Google Cloud storage bucket.

@ -109,7 +82,32 @@ Create a Compute Engine image from the file.
 gcloud compute images create flatcar-linux-2303-4-0 --source-uri gs://BUCKET_NAME/flatcar_production_gce.tar.gz
 ```

-Set the [os_image](#variables) to the image name (e.g. `flatcar-linux-2303-4-0`)
+Set the [os_image](#variables) in the next step.
+
+## Cluster
+
+Define a Kubernetes cluster using the module `google-cloud/container-linux/kubernetes`.
+
+```tf
+module "yavin" {
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.2"
+
+  # Google Cloud
+  cluster_name  = "yavin"
+  region        = "us-central1"
+  dns_zone      = "example.com"
+  dns_zone_name = "example-zone"
+
+  # configuration
+  os_image           = "flatcar-linux-2303-4-0"
+  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+
+  # optional
+  worker_count = 2
+}
+```
+
+Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.

 ## ssh-agent

@ -167,9 +165,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
 ```

 List the pods.
@ -206,6 +204,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-
 | region | Google Cloud region | "us-central1" |
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
+| os_image | Container Linux image for compute instances | "flatcar-linux-2303-4-0", coreos-stable, coreos-beta, coreos-alpha |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

 Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Container Linux [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep coreos`.
@ -235,7 +234,6 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
-| os_image | Container Linux image for compute instances | "coreos-stable" | "flatcar-linux-2303-4-0" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.18.2"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.18.1
-ip-10-0-26-65  Ready   <none>   10m  v1.18.1
-ip-10-0-41-21  Ready   <none>   10m  v1.18.1
+ip-10-0-3-155  Ready   <none>   10m  v1.18.2
+ip-10-0-26-65  Ready   <none>   10m  v1.18.2
+ip-10-0-41-21  Ready   <none>   10m  v1.18.2
 ```

 List the pods.
--- a/docs/fedora-coreos/azure.md
+++ b/docs/fedora-coreos/azure.md
@ -0,0 +1,261 @@
+# Azure
+
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Azure with Fedora CoreOS.
+
+We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
+
+Controller hosts are provisioned to run an `etcd-member` peer and a `kubelet` service. Worker hosts run a `kubelet` service. Controller nodes run `kube-apiserver`, `kube-scheduler`, `kube-controller-manager`, and `coredns`, while `kube-proxy` and `calico` (or `flannel`) run on every node. A generated `kubeconfig` provides `kubectl` access to the cluster.
+
+## Requirements
+
+* Azure account
+* Azure DNS Zone (registered Domain Name or delegated subdomain)
+* Terraform v0.12.6+ and [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) installed locally
+
+## Terraform Setup
+
+Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your system.
+
+```sh
+$ terraform version
+Terraform v0.12.21
+```
+
+Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
+
+```sh
+wget https://github.com/poseidon/terraform-provider-ct/releases/download/v0.5.0/terraform-provider-ct-v0.5.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.5.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.5.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.5.0
+```
+
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+
+```
+cd infra/clusters
+```
+
+## Provider
+
+[Install](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) the Azure `az` command line tool to [authenticate with Azure](https://www.terraform.io/docs/providers/azurerm/authenticating_via_azure_cli.html).
+
+```
+az login
+```
+
+Configure the Azure provider in a `providers.tf` file.
+
+```tf
+provider "azurerm" {
+  version = "2.5.0"
+}
+
+provider "ct" {
+  version = "0.5.0"
+}
+```
+
+Additional configuration options are described in the `azurerm` provider [docs](https://www.terraform.io/docs/providers/azurerm/).
+
+## Fedora CoreOS Images
+
+Fedora CoreOS publishes images for Azure, but does not yet upload them. Azure allows custom images to be uploaded to a storage account bucket and imported.
+
+[Download](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) a Fedora CoreOS Azure VHD image and upload it to an Azure storage account container (i.e. bucket) via the UI (quite slow).
+
+```
+xz -d fedora-coreos-31.20200323.3.2-azure.x86_64.vhd.xz
+```
+
+Create an Azure disk (note disk ID) and create an Azure image from it (note image ID).
+
+```
+az disk create --name fedora-coreos-31.20200323.3.2 -g GROUP --source https://BUCKET.blob.core.windows.net/fedora-coreos/fedora-coreos-31.20200323.3.2-azure.x86_64.vhd
+
+az image create --name fedora-coreos-31.20200323.3.2 -g GROUP --os-type=linux --source /subscriptions/some/path/providers/Microsoft.Compute/disks/fedora-coreos-31.20200323.3.2
+```
+
+Set the [os_image](#variables) in the next step.
+
+## Cluster
+
+Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.
+
+```tf
+module "ramius" {
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.18.2"
+
+  # Azure
+  cluster_name   = "ramius"
+  region         = "centralus"
+  dns_zone       = "azure.example.com"
+  dns_zone_group = "example-group"
+
+  # configuration
+  os_image           = "/subscriptions/some/path/Microsoft.Compute/images/fedora-coreos-31.20200323.3.2"
+  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+
+  # optional
+  worker_count    = 2
+  host_cidr       = "10.0.0.0/20"
+}
+```
+
+Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/fedora-coreos/kubernetes/variables.tf) source.
+
+## ssh-agent
+
+Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
+
+```sh
+ssh-add ~/.ssh/id_rsa
+ssh-add -L
+```
+
+## Apply
+
+Initialize the config directory if this is the first use with Terraform.
+
+```sh
+terraform init
+```
+
+Plan the resources to be created.
+
+```sh
+$ terraform plan
+Plan: 86 to add, 0 to change, 0 to destroy.
+```
+
+Apply the changes to create the cluster.
+
+```sh
+$ terraform apply
+...
+module.ramius.null_resource.bootstrap: Still creating... (6m50s elapsed)
+module.ramius.null_resource.bootstrap: Still creating... (7m0s elapsed)
+module.ramius.null_resource.bootstrap: Creation complete after 7m8s (ID: 3961816482286168143)
+
+Apply complete! Resources: 69 added, 0 changed, 0 destroyed.
+```
+
+In 4-8 minutes, the Kubernetes cluster will be ready.
+
+## Verify
+
+[Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your system. Obtain the generated cluster `kubeconfig` from module outputs (e.g. write to a local file).
+
+```
+resource "local_file" "kubeconfig-ramius" {
+  content  = module.ramius.kubeconfig-admin
+  filename = "/home/user/.kube/configs/ramius-config"
+}
+```
+
+List nodes in the cluster.
+
+```
+$ export KUBECONFIG=/home/user/.kube/configs/ramius-config
+$ kubectl get nodes
+NAME                  STATUS  ROLES   AGE  VERSION
+ramius-controller-0   Ready   <none>  24m  v1.18.2
+ramius-worker-000001  Ready   <none>  25m  v1.18.2
+ramius-worker-000002  Ready   <none>  24m  v1.18.2
+```
+
+List the pods.
+
+```
+$ kubectl get pods --all-namespaces
+NAMESPACE     NAME                                        READY  STATUS    RESTARTS  AGE
+kube-system   coredns-7c6fbb4f4b-b6qzx                    1/1    Running   0         26m
+kube-system   coredns-7c6fbb4f4b-j2k3d                    1/1    Running   0         26m
+kube-system   calico-node-1m5bf                           2/2    Running   0         26m
+kube-system   calico-node-7jmr1                           2/2    Running   0         26m
+kube-system   calico-node-bknc8                           2/2    Running   0         26m
+kube-system   kube-apiserver-ramius-controller-0          1/1    Running   0         26m
+kube-system   kube-controller-manager-ramius-controller-0 1/1    Running   0         26m
+kube-system   kube-proxy-j4vpq                            1/1    Running   0         26m
+kube-system   kube-proxy-jxr5d                            1/1    Running   0         26m
+kube-system   kube-proxy-lbdw5                            1/1    Running   0         26m
+kube-system   kube-scheduler-ramius-controller-0          1/1    Running   0         26m
+```
+
+## Going Further
+
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
+
+## Variables
+
+Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/fedora-coreos/kubernetes/variables.tf) source.
+
+### Required
+
+| Name | Description | Example |
+|:-----|:------------|:--------|
+| cluster_name | Unique cluster name (prepended to dns_zone) | "ramius" |
+| region | Azure region | "centralus" |
+| dns_zone | Azure DNS zone | "azure.example.com" |
+| dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
+| os_image | Fedora CoreOS image for instances | "/subscriptions/..../custom-image" |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+
+!!! tip
+    Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
+
+#### DNS Zone
+
+Clusters create a DNS A record `${cluster_name}.${dns_zone}` to resolve a load balancer backed by controller instances. This FQDN is used by workers and `kubectl` to access the apiserver(s). In this example, the cluster's apiserver would be accessible at `ramius.azure.example.com`.
+
+You'll need a registered domain name or delegated subdomain on Azure DNS. You can set this up once and create many clusters with unique names.
+
+```tf
+# Azure resource group for DNS zone
+resource "azurerm_resource_group" "global" {
+  name     = "global"
+  location = "centralus"
+}
+
+# DNS zone for clusters
+resource "azurerm_dns_zone" "clusters" {
+  resource_group_name = azurerm_resource_group.global.name
+
+  name      = "azure.example.com"
+  zone_type = "Public"
+}
+```
+
+Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource group with `"azurerm_resource_group.global.name`.
+
+!!! tip ""
+    If you have an existing domain name with a zone file elsewhere, just delegate a subdomain that can be managed on Azure DNS (e.g. azure.mydomain.com) and [update nameservers](https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns).
+
+### Optional
+
+| Name | Description | Default | Example |
+|:-----|:------------|:--------|:--------|
+| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
+| worker_count | Number of workers | 1 | 3 |
+| controller_type | Machine type for controllers | "Standard_B2s" | See below |
+| worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
+| disk_size | Size of the disk in GB | 40 | 100 |
+| worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
+| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
+| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
+| networking | Choice of networking provider | "calico" | "flannel" or "calico" |
+| host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
+| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
+| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| worker_node_labels | List of initial worker node labels | [] | ["worker-pool=default"] |
+
+Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
+
+!!! warning
+    Unlike AWS and GCP, Azure requires its *virtual* networks to have non-overlapping IPv4 CIDRs (yeah, go figure). Instead of each cluster just using `10.0.0.0/16` for instances, each Azure cluster's `host_cidr` must be non-overlapping (e.g. 10.0.0.0/20 for the 1st cluster, 10.0.16.0/20 for the 2nd cluster, etc).
+
+!!! warning
+    Do not choose a `controller_type` smaller than `Standard_B2s`. Smaller instances are not sufficient for running a controller.
+
+#### Spot Priority
+
+Add `worker_priority=Spot` to use [Spot Priority](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/spot-vms) workers that run on Azure's surplus capacity at lower cost, but with the tradeoff that they can be deallocated at random. Spot priority VMs are Azure's analog to AWS spot instances or GCP premptible instances.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.18.1 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.18.2 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.2"

  # bare-metal
  cluster_name            = "mercury"
@ -289,9 +289,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.18.1
-node2.example.com   Ready   <none>  10m  v1.18.1
-node3.example.com   Ready   <none>  10m  v1.18.1
+node1.example.com   Ready   <none>  10m  v1.18.2
+node2.example.com   Ready   <none>  10m  v1.18.2
+node3.example.com   Ready   <none>  10m  v1.18.2
 ```

 List the pods.
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on DigitalOcean with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -63,7 +63,7 @@ provider "ct" {

 Fedora CoreOS publishes images for DigitalOcean, but does not yet upload them. DigitalOcean allows [custom images](https://blog.digitalocean.com/custom-images/) to be uploaded via URL or file.

-Import a [Fedora CoreOS](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) image via URL to desired a region(s). Reference the DigitalOcean image and set the `os_image` in the next step.
+Import a [Fedora CoreOS](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) image via URL to desired a region(s).

 ```tf
 data "digitalocean_image" "fedora-coreos-31-20200323-3-2" {
@ -71,21 +71,23 @@ data "digitalocean_image" "fedora-coreos-31-20200323-3-2" {
 }
 ```

+Set the [os_image](#variables) in the next step.
+
 ## Cluster

 Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubernetes`.

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.18.2"

  # Digital Ocean
  cluster_name = "nemo"
  region       = "nyc3"
  dns_zone     = "digital-ocean.example.com"
-  os_image     = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id

  # configuration
+  os_image         = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id
  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]

  # optional
@ -151,9 +153,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.18.1
-10.132.115.81      Ready   <none>  10m  v1.18.1
-10.132.124.107     Ready   <none>  10m  v1.18.1
+10.132.110.130     Ready   <none>  10m  v1.18.2
+10.132.115.81      Ready   <none>  10m  v1.18.2
+10.132.124.107     Ready   <none>  10m  v1.18.2
 ```

 List the pods.
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.18.1 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Google Compute Engine with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -82,6 +82,8 @@ Create a Compute Engine image from the file.
 gcloud compute images create fedora-coreos-31-20200323-3-2 --source-uri gs://BUCKET/fedora-coreos-31.20200323.3.2-gcp.x86_64.tar.gz
 ```

+Set the [os_image](#variables) in the next step.
+
 ## Cluster

 Define a Kubernetes cluster using the module `google-cloud/fedora-coreos/kubernetes`.
@ -96,10 +98,8 @@ module "yavin" {
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"

-  # custom image name from above
-  os_image = "fedora-coreos-31-20200323-3-2"
-
  # configuration
+  os_image           = "fedora-coreos-31-20200323-3-2"
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."

  # optional
@ -165,9 +165,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
 ```

 List the pods.
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
@ -26,6 +26,7 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | stable |
+| Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](fedora-coreos/azure.md) | alpha |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | beta |
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](fedora-coreos/digitalocean.md) | alpha |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |
@ -54,7 +55,7 @@ Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com
 ## Documentation

 * Architecture [concepts](architecture/concepts.md) and [operating-systems](architecture/operating-systems.md)
-* Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Bare-Metal](fedora-coreos/bare-metal.md), [DigitalOcean](fedora-coreos/digitalocean.md), and [Google Cloud](fedora-coreos/google-cloud.md)
+* Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Azure](fedora-coreos/azure.md), [Bare-Metal](fedora-coreos/bare-metal.md), [DigitalOcean](fedora-coreos/digitalocean.md), and [Google Cloud](fedora-coreos/google-cloud.md)
 * Flatcar Linux tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [DigitalOcean](cl/digital-ocean.md), and [Google Cloud](cl/google-cloud.md)

 ## Example
@ -63,7 +64,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.2"

  # Google Cloud
  cluster_name  = "yavin"
@ -101,9 +102,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
 ```

 List the pods.
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -18,7 +18,7 @@ module "yavin" {
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.2"
  ...
 }
 ```
@ -279,15 +279,15 @@ Typhoon modules have been adapted for Terraform v0.12. Provider plugins requirem

 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.18.1 - ?       | v0.12.x             |
-| v1.10.3 - v1.18.1 | v0.11.x             |
+| v1.18.2 - ?       | v0.12.x             |
+| v1.10.3 - v1.18.2 | v0.11.x             |
 | v1.9.2 - v1.10.2  | v0.10.4+ or v0.11.x |
 | v1.7.3 - v1.9.1   | v0.10.x             |
 | v1.6.4 - v1.7.2   | v0.9.x              |

 ### New users

-New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.18.1+ without issue.
+New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.18.2+ without issue.

 ### Existing users

--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/google-cloud/container-linux/kubernetes/bootstrap.tf
+++ b/google-cloud/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/container-linux/kubernetes/cl/controller.yaml
+++ b/google-cloud/container-linux/kubernetes/cl/controller.yaml
@ -90,7 +90,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -167,8 +167,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -182,10 +180,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/google-cloud/container-linux/kubernetes/variables.tf
+++ b/google-cloud/container-linux/kubernetes/variables.tf
@ -48,8 +48,7 @@ variable "worker_type" {

 variable "os_image" {
  type        = string
-  description = "Container Linux image for compute instances (e.g. coreos-stable)"
-  default     = "coreos-stable"
+  description = "Container Linux image for compute instances (e.g. coreos-stable, custom-image)"
 }

 variable "disk_size" {
@ -83,12 +82,6 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (flannel or calico)"
@ -131,6 +124,12 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml
@ -63,7 +63,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -125,7 +125,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.1 \
+            docker://quay.io/poseidon/kubelet:v1.18.2 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/google-cloud/container-linux/kubernetes/workers/variables.tf
+++ b/google-cloud/container-linux/kubernetes/workers/variables.tf
@ -37,7 +37,6 @@ variable "machine_type" {
 variable "os_image" {
  type        = string
  description = "Container Linux image for compute instanges (e.g. gcloud compute images list)"
-  default     = "coreos-stable"
 }

 variable "disk_size" {
--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@ -11,13 +11,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.1 (upstream)
+* Kubernetes v1.18.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

-Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/cl/google-cloud/).
+Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/fedora-coreos/google-cloud/).

--- a/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
+++ b/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
@ -79,7 +79,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -119,11 +119,11 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.1
+            quay.io/poseidon/kubelet:v1.18.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -155,8 +155,6 @@ storage:
          sudo mv static-manifests/* /etc/kubernetes/manifests/
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
@ -169,10 +167,6 @@ storage:
            echo "Waiting for static pod control plane"
            sleep 5
          done
-          until kubectl apply -f /assets/manifests/crds -R; do
-             echo "Retry Custom Resource Definition manifests"
-             sleep 5
-          done
          until kubectl apply -f /assets/manifests -R; do
             echo "Retry applying manifests"
             sleep 5
--- a/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -49,7 +49,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.1 \
+          quay.io/poseidon/kubelet:v1.18.2 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -87,7 +87,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/mkdocs.yml
+++ b/mkdocs.yml
@ -56,6 +56,7 @@ nav:
  - 'Fedora CoreOS':
    - 'AWS': 'fedora-coreos/aws.md'
    - 'Bare-Metal': 'fedora-coreos/bare-metal.md'
+    - 'Digital Ocean': 'fedora-coreos/digitalocean.md'
    - 'Google Cloud': 'fedora-coreos/google-cloud.md'
  - 'Container Linux':
    - 'AWS': 'cl/aws.md'
Author	SHA1	Message	Date
Dalton Hubble	fcbee12334	Fix race condition creating DigitalOcean firewall rules * DigitalOcean firewall rules should reference Terraform tag resources rather than using tag strings. Otherwise, terraform apply can fail (neeeds rerun) if a tag has not yet been created	2020-04-19 16:55:02 -07:00
Dalton Hubble	feac94605a	Fix bootstrap mount to use shared volume SELinux label * Race: During initial bootstrap, static control plane pods could hang with Permission denied to bootstrap secrets. A manual fix involved restarting Kubelet, which relabeled mounts The race had no effect on subsequent reboots. * bootstrap.service runs podman with a private unshared mount of /etc/kubernetes/bootstrap-secrets which uses an SELinux MCS label with a category pair. However, bootstrap-secrets should be shared as its mounted by Docker pods kube-apiserver, kube-scheduler, and kube-controller-manager. Restarting Kubelet was a manual fix because Kubelet relabels all /etc/kubernetes * Fix bootstrap Pod to use the shared volume label, which leaves bootstrap-secrets files with SELinux level s0 without MCS * Also allow failed bootstrap.service to be re-applied. This was missing on bare-metal and AWS	2020-04-19 16:31:32 -07:00
Dalton Hubble	2b1b918b43	Revert Flatcar Linux Azure to manual upload images * Initial support for Flatcar Linux on Azure used the Flatcar Linux Azure Marketplace images (e.g. `flatcar-stable`) in https://github.com/poseidon/typhoon/pull/664 * Flatcar Linux Azure Marketplace images have some unresolved items https://github.com/poseidon/typhoon/issues/703 * Until the Marketplace items are resolved, revert to requiring Flatcar Linux's images be manually uploaded (like GCP and DigitalOcean)	2020-04-18 15:40:57 -07:00
Dalton Hubble	bf22222f7d	Remove temporary workaround for v1.18.0 apply issue * In v1.18.0, kubectl apply would fail to apply manifests if any single manifest was unable to validate. For example, if a CRD and CR were defined in the same directory, apply would fail since the CR would be invalid as the CRD wouldn't exist * Typhoon temporary workaround was to separate CNI CRD manifests and explicitly apply them first. No longer needed in v1.18.1+ * Kubernetes v1.18.1 restored the prior behavior where kubectl apply applies as many valid manifests as it can. In the example above, the CRD would be applied and the CR could be applied if the kubectl apply was re-run (allowing for apply loops). * Upstream fix: https://github.com/kubernetes/kubernetes/pull/89864	2020-04-16 23:49:55 -07:00
Dalton Hubble	671eacb86e	Update Kubernetes from v1.18.1 to v1.18.2 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#changelog-since-v1181	2020-04-16 23:40:52 -07:00
Dalton Hubble	e2d4af43be	Fix Fedora CoreOS Azure MTU with Calico * With Calico VXLAN on Fedora CoreOS the 1450 MTU should be used	2020-04-12 23:20:04 -07:00
Dalton Hubble	5c4a3f73d5	Add support for Fedora CoreOS on Azure * Add `azure/fedora-coreos/kubernetes` module	2020-04-12 16:35:49 -07:00
Dalton Hubble	76ab4c4c2a	Change `container-linux` module preference to Flatcar Linux * No change to Fedora CoreOS modules * For Container Linx AWS and Azure, change the `os_image` default from coreos-stable to flatcar-stable * For Container Linux GCP and DigitalOcean, change `os_image` to be required since users should upload a Flatcar Linux image and set the variable * For Container Linux bare-metal, recommend users change the `os_channel` to Flatcar Linux. No actual module change.	2020-04-11 14:52:30 -07:00
Dalton Hubble	1627ecaf27	Fix docs TOC to include Fedora CoreOS DigitalOcean	2020-04-11 14:07:09 -07:00