Update docs and README for release

Remove iSCSI mounts from Kubelet
* Remove Kubelet `/etc/iscsi` and `iscsiadm` host mounts that were added on bare-metal, since these no longer work on either Fedora CoreOS or Flatcar Linux with newer `iscsiadm` * These special mounts on bare-metal date back to #350 which added them to provide a way to use iSCSI in Kubernetes v1.10 * Today, storage should be handled by external CSI providers which handle different storage systems, which doesn't rely on Kubelet storage utils Close #907
2025-08-02 19:01:34 +02:00 · 2020-12-12 12:31:35 -08:00 · 2020-12-12 11:41:02 -08:00 · 2020-12-11 00:47:57 -08:00 · 2020-12-11 00:24:43 -08:00 · 2020-12-11 00:15:29 -08:00
172 changed files with 2454 additions and 2261 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -2,10 +2,94 @@

 Notable changes between versions.

-## Latest
+## v1.20.0
+
+* Kubernetes [v1.20.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1200)
+* Add input variable validations ([#880](https://github.com/poseidon/typhoon/pull/880))
+  * Require Terraform v0.13+ ([migration guide](https://typhoon.psdn.io/topics/maintenance/#terraform-versions))
+* Set output sensitive to suppress console display for some cases ([#885](https://github.com/poseidon/typhoon/pull/885))
+* Add service account token [volume projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection) ([#897](https://github.com/poseidon/typhoon/pull/897))
+* Scope kube-scheduler and kube-controller-manager permissions ([#898](https://github.com/poseidon/typhoon/pull/898))
+* Update etcd from v3.4.12 to [v3.4.14](https://github.com/etcd-io/etcd/releases/tag/v3.4.14)
+* Update Calico from v3.16.5 to v3.17.1 ([#890](https://github.com/poseidon/typhoon/pull/890))
+  * Enable Calico MTU auto-detection
+  * Remove [workaround](https://github.com/poseidon/typhoon/pull/724) to Calico cni-plugin [issue](https://github.com/projectcalico/cni-plugin/issues/874)
+* Update Cilium from v1.9.0 to [v1.9.1](https://github.com/cilium/cilium/releases/tag/v1.9.1)
+* Relax `terraform-provider-ct` version constraint to v0.6+ ([#893](https://github.com/poseidon/typhoon/pull/893))
+  * Allow upgrading `terraform-provider-ct` to v0.7.x ([warn](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+
+### AWS
+
+* Enable Network Load Balancer (NLB) dualstack ([#883](https://github.com/poseidon/typhoon/pull/883))
+  * NLB subnets assigned both IPv4 and IPv6 addresses
+  * NLB DNS name has both A and AAAA records
+  * NLB to target node traffic is IPv4 (no change)
+
+### Bare-Metal
+
+* Remove iSCSI `/etc/iscsi` and `iscsadm` mounts from Kubelet ([#912](https://github.com/poseidon/typhoon/pull/912))
+
+### Fedora CoreOS
+
+#### AWS
+
+* Fix AMI query for which could fail in some regions ([#887](https://github.com/poseidon/typhoon/pull/887))
+
+#### Bare-Metal
+
+* Promote Fedora CoreOS to stable
+* Use initramfs and rootfs images as initrd's ([#889](https://github.com/poseidon/typhoon/pull/889))
+  * Requires Fedora CoreOS version with rootfs images (e.g. 32.20200923.3.0+)
+
+### Addons
+
+* Update Prometheus from v2.22.2 to [v2.23.0](https://github.com/prometheus/prometheus/releases/tag/v2.23.0)
+* Update kube-state-metrics from v2.0.0-alpha.2 to [v2.0.0-alpha.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.3)
+* Update Grafana from v7.3.2 to [v7.3.5](https://github.com/grafana/grafana/releases/tag/v7.3.5)
+
+## v1.19.4
+
+* Kubernetes [v1.19.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1194)
+* Update Cilium from v1.8.4 to [v1.9.0](https://github.com/cilium/cilium/releases/tag/v1.9.0)
+* Update Calico from v3.16.3 to [v3.16.5](https://github.com/projectcalico/calico/releases/tag/v3.16.5)
+* Remove `asset_dir` variable (defaulted off in [v1.17.0](https://github.com/poseidon/typhoon/pull/595), deprecated in [v1.18.0](https://github.com/poseidon/typhoon/pull/678))
+
+### Fedora CoreOS
+
+* Improve `etcd-member.service` systemd unit ([#868](https://github.com/poseidon/typhoon/pull/868))
+  * Allow a snippet with a systemd dropin to set an alternate image (e.g. mirror)
+* Fix local node delete oneshot on node shutdown ([#856](https://github.com/poseidon/typhoon/pull/855))
+
+#### AWS
+
+* Add experimental Fedora CoreOS arm64 support ([docs](https://typhoon.psdn.io/advanced/arm64/), [#875](https://github.com/poseidon/typhoon/pull/875))
+  * Allow arm64 full-cluster or mixed/hybrid cluster with worker pools
+  * Add `arch` variable to cluster module
+  * Add `daemonset_tolerations` variable to cluster module
+  * Add `node_taints` variable to workers module
+  * Requires flannel CNI provider and use of experimental AMI (see docs)
+
+### Flatcar Linux
+
+* Rename `container-linux` modules to `flatcar-linux` ([#858](https://github.com/poseidon/typhoon/issues/858)) (**action required**)
+* Change on-host system containers from rkt to docker
+  * Change `etcd-member.service` container runnner from rkt to docker ([#867](https://github.com/poseidon/typhoon/pull/867))
+  * Change `kubelet.service` container runner from rkt-fly to docker ([#855](https://github.com/poseidon/typhoon/pull/855))
+  * Change `bootstrap.service` container runner from rkt to docker ([#873](https://github.com/poseidon/typhoon/pull/873))
+  * Change `delete-node.service` to use docker and an inline ExecStart ([#855](https://github.com/poseidon/typhoon/pull/855))
+* Fix local node delete oneshot on node shutdown ([#855](https://github.com/poseidon/typhoon/pull/855))
+* Remove CoreOS Container Linux Matchbox profiles ([#859](https://github.com/poseidon/typhoon/pull/858))
+
+### Addons
+
+* Update nginx-ingress from v0.40.2 to [v0.41.2](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.41.2)
+* Update Prometheus from v2.22.0 to [v2.22.1](https://github.com/prometheus/prometheus/releases/tag/v2.22.1)
+* Update kube-state-metrics from v2.0.0-alpha.1 to [v2.0.0-alpha.2](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.2)
+* Update Grafana from v7.2.1 to [v7.3.2](https://github.com/grafana/grafana/releases/tag/v7.3.2)

 ## v1.19.3

+* Kubernetes [v1.19.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1193)
 * Update Cilium from v1.8.3 to [v1.8.4](https://github.com/cilium/cilium/releases/tag/v1.8.4)
 * Update Calico from v1.15.3 to [v1.16.3](https://github.com/projectcalico/calico/releases/tag/v3.16.3) ([#851](https://github.com/poseidon/typhoon/pull/851))
 * Update flannel from v0.13.0-rc2 to v0.13.0 ([#219](https://github.com/poseidon/terraform-render-bootstrap/pull/219))
--- a/README.md
+++ b/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)

 ## Modules
@ -27,7 +27,7 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 |---------------|------------------|------------------|--------|
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | stable |
 | Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](azure/fedora-coreos/kubernetes) | alpha |
-| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
+| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | stable |
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | stable |

@ -35,11 +35,11 @@ Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
-| Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | beta |
+| AWS           | Flatcar Linux    | [aws/flatcar-linux/kubernetes](aws/flatcar-linux/kubernetes) | stable |
+| Azure         | Flatcar Linux    | [azure/flatcar-linux/kubernetes](azure/flatcar-linux/kubernetes) | alpha |
+| Bare-Metal    | Flatcar Linux    | [bare-metal/flatcar-linux/kubernetes](bare-metal/flatcar-linux/kubernetes) | stable |
+| DigitalOcean | Flatcar Linux  | [digital-ocean/flatcar-linux/kubernetes](digital-ocean/flatcar-linux/kubernetes) | beta |
+| Google Cloud  | Flatcar Linux  | [google-cloud/flatcar-linux/kubernetes](google-cloud/flatcar-linux/kubernetes) | beta |

 ## Documentation

@ -54,7 +54,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.19.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.20.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -93,9 +93,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.19.3
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.19.3
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.19.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.0
 ```

 List the pods.
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -24,7 +24,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:7.2.1
+          image: docker.io/grafana/grafana:7.3.5
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.40.2
+          image: k8s.gcr.io/ingress-nginx/controller:v0.41.2
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.40.2
+          image: k8s.gcr.io/ingress-nginx/controller:v0.41.2
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.40.2
+          image: k8s.gcr.io/ingress-nginx/controller:v0.41.2
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.40.2
+          image: k8s.gcr.io/ingress-nginx/controller:v0.41.2
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.40.2
+          image: k8s.gcr.io/ingress-nginx/controller:v0.41.2
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -21,7 +21,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.22.0
+          image: quay.io/prometheus/prometheus:v2.23.0
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
@ -90,6 +90,7 @@ rules:
  - networking.k8s.io
  resources:
  - networkpolicies
+  - ingresses
  verbs:
  - list
  - watch
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -25,7 +25,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v2.0.0-alpha.1
+        image: quay.io/coreos/kube-state-metrics:v2.0.0-alpha.3
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -50,28 +50,6 @@ data:
                "severity": "warning"
              }
            },
-            {
-              "alert": "etcdHighNumberOfFailedGRPCRequests",
-              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}."
-              },
-              "expr": "100 * sum(rate(grpc_server_handled_total{job=~\".*etcd.*\", grpc_code!=\"OK\"}[5m])) without (grpc_type, grpc_code)\n  /\nsum(rate(grpc_server_handled_total{job=~\".*etcd.*\"}[5m])) without (grpc_type, grpc_code)\n  > 1\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
-            {
-              "alert": "etcdHighNumberOfFailedGRPCRequests",
-              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}."
-              },
-              "expr": "100 * sum(rate(grpc_server_handled_total{job=~\".*etcd.*\", grpc_code!=\"OK\"}[5m])) without (grpc_type, grpc_code)\n  /\nsum(rate(grpc_server_handled_total{job=~\".*etcd.*\"}[5m])) without (grpc_type, grpc_code)\n  > 5\n",
-              "for": "5m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
            {
              "alert": "etcdGRPCRequestsSlow",
              "annotations": {
--- a/aws/container-linux/kubernetes/cl/controller.yaml
+++ b/aws/container-linux/kubernetes/cl/controller.yaml
@ -1,205 +0,0 @@
---
-systemd:
-  units:
-    - name: etcd-member.service
-      enabled: true
-      dropins:
-        - name: 40-etcd-cluster.conf
-          contents: |
-            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.12"
-            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
-            Environment="RKT_RUN_ARGS=--insecure-options=image"
-            Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
-            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
-            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
-            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
-            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
-            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
-            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
-            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
-            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
-            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
-            Environment="ETCD_CLIENT_CERT_AUTH=true"
-            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
-            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
-            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
-            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-        RequiredBy=etcd-member.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=10
-        [Install]
-        WantedBy=multi-user.target
-    - name: bootstrap.service
-      contents: |
-        [Unit]
-        Description=Kubernetes control plane
-        ConditionPathExists=!/opt/bootstrap/bootstrap.done
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        WorkingDirectory=/opt/bootstrap
-        ExecStart=/usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes/bootstrap-secrets \
-            --mount volume=config,target=/etc/kubernetes/secrets \
-            --volume assets,kind=host,source=/opt/bootstrap/assets \
-            --mount volume=assets,target=/assets \
-            --volume script,kind=host,source=/opt/bootstrap/apply \
-            --mount volume=script,target=/apply \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.19.3 \
-            --net=host \
-            --dns=host \
-            --exec=/apply
-        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  directories:
-    - path: /var/lib/etcd
-      filesystem: root
-      mode: 0700
-      overwrite: true
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /opt/bootstrap/layout
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
-          awk '/#####/ {filename=$2; next} {print > filename}' assets
-          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
-          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
-          chown -R etcd:etcd /etc/ssl/etcd
-          chmod -R 500 /etc/ssl/etcd
-          chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          mkdir -p /etc/kubernetes/manifests
-          mv static-manifests/* /etc/kubernetes/manifests/
-          mkdir -p /opt/bootstrap/assets
-          mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls manifests-networking
-    - path: /opt/bootstrap/apply
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
-          until kubectl version; do
-            echo "Waiting for static pod control plane"
-            sleep 5
-          done
-          until kubectl apply -f /assets/manifests -R; do
-             echo "Retry applying manifests"
-             sleep 5
-          done
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml
@ -1,140 +0,0 @@
---
-systemd:
-  units:
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/node \
-          %{~ for label in split(",", node_labels) ~}
-          --node-labels=${label} \
-          %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=5
-        [Install]
-        WantedBy=multi-user.target
-    - name: delete-node.service
-      enable: true
-      contents: |
-        [Unit]
-        Description=Waiting to delete Kubernetes node on shutdown
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/etc/kubernetes/delete-node
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-    - path: /etc/kubernetes/delete-node
-      filesystem: root
-      mode: 0744
-      contents:
-        inline: |
-          #!/bin/bash
-          set -e
-          exec /usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes \
-            --mount volume=config,target=/etc/kubernetes \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.19.3 \
-            --net=host \
-            --dns=host \
-            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -18,3 +18,29 @@ data "aws_ami" "fedora-coreos" {
    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
+
+# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
+# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
+# and may be removed for any reason before then as well. Do not use.
+data "aws_ami" "fedora-coreos-arm" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["099663496933"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["fedora-coreos-*"]
+  }
+}
+
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=7988fb7159cb81e2d080b365b147fe90542fd258"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4edd79dd0295e6ffa4c8ed04fd5914d5cb3f1b7c"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers          = aws_route53_record.etcds.*.fqdn
-  asset_dir             = var.asset_dir
  networking            = var.networking
  network_mtu           = var.network_mtu
  pod_cidr              = var.pod_cidr
@ -13,6 +12,7 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations

  trusted_certs_dir = "/etc/pki/tls/certs"
 }
--- a/aws/fedora-coreos/kubernetes/controllers.tf
+++ b/aws/fedora-coreos/kubernetes/controllers.tf
@ -22,9 +22,8 @@ resource "aws_instance" "controllers" {
  }

  instance_type = var.controller_type
-
-  ami       = data.aws_ami.fedora-coreos.image_id
-  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]
+  ami           = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+  user_data     = data.ct_config.controller-ignitions.*.rendered[count.index]

  # storage
  root_block_device {
@ -63,6 +62,7 @@ data "template_file" "controller-configs" {

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_arch   = var.arch == "arm64" ? "-arm64" : ""
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -8,28 +8,25 @@ systemd:
      contents: |
        [Unit]
        Description=etcd (System Container)
-        Documentation=https://github.com/coreos/etcd
+        Documentation=https://github.com/etcd-io/etcd
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        # https://github.com/opencontainers/runc/pull/1807
-        # Type=notify
-        # NotifyAccess=exec
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14${etcd_arch}
        Type=exec
-        Restart=on-failure
-        RestartSec=10s
-        TimeoutStartSec=0
-        LimitNOFILE=40000
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
-        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
        ExecStart=/usr/bin/podman run --name etcd \
          --env-file /etc/etcd/etcd.env \
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.12
+          $${ETCD_IMAGE}
        ExecStop=/usr/bin/podman stop etcd
+        Restart=on-failure
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
    - name: docker.service
@ -55,7 +52,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -69,12 +66,10 @@ systemd:
          --network host \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
          --volume /var/lib/calico:/var/lib/calico:ro \
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@ -120,11 +115,11 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
+            --volume /etc/kubernetes/pki:/etc/kubernetes/pki:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.19.3
+            quay.io/poseidon/kubelet:v1.20.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -147,26 +142,26 @@ storage:
          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
          awk '/#####/ {filename=$2; next} {print > filename}' assets
          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mkdir -p /etc/kubernetes/pki
          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
          chown -R etcd:etcd /etc/ssl/etcd
          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
          mkdir -p /etc/kubernetes/manifests
          mv static-manifests/* /etc/kubernetes/manifests/
          mkdir -p /opt/bootstrap/assets
          mv manifests /opt/bootstrap/assets/manifests
          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
-          chcon -R -u system_u -t container_file_t /etc/kubernetes/bootstrap-secrets
+          chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
        inline: |
          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
          until kubectl version; do
            echo "Waiting for static pod control plane"
            sleep 5
@ -202,8 +197,6 @@ storage:
      mode: 0644
      contents:
        inline: |
-          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
-          NOTIFY_SOCKET=/run/systemd/notify
          ETCD_NAME=${etcd_name}
          ETCD_DATA_DIR=/var/lib/etcd
          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
@ -221,6 +214,7 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+          ETCD_UNSUPPORTED_ARCH=arm64
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/nlb.tf
+++ b/aws/fedora-coreos/kubernetes/nlb.tf
@ -17,6 +17,7 @@ resource "aws_route53_record" "apiserver" {
 resource "aws_lb" "nlb" {
  name               = "${var.cluster_name}-nlb"
  load_balancer_type = "network"
+  ip_address_type    = "dualstack"
  internal           = false

  subnets = aws_subnet.public.*.id
--- a/aws/fedora-coreos/kubernetes/outputs.tf
+++ b/aws/fedora-coreos/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "worker_security_groups" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom load balancing
@ -52,3 +54,10 @@ output "worker_target_group_https" {
  value       = module.workers.target_group_https
 }

+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -43,8 +43,13 @@ variable "worker_type" {

 variable "os_stream" {
  type        = string
-  description = "Fedora CoreOs image stream for instances (e.g. stable, testing, next)"
+  description = "Fedora CoreOS image stream for instances (e.g. stable, testing, next)"
  default     = "stable"
+
+  validation {
+    condition     = contains(["stable", "testing", "next"], var.os_stream)
+    error_message = "The os_stream must be stable, testing, or next."
+  }
 }

 variable "disk_size" {
@ -96,12 +101,6 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (calico or flannel)"
@ -161,3 +160,20 @@ variable "cluster_domain_suffix" {
  default     = "cluster.local"
 }

+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
+
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
+
--- a/aws/fedora-coreos/kubernetes/versions.tf
+++ b/aws/fedora-coreos/kubernetes/versions.tf
@ -1,7 +1,7 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
    template = "~> 2.1"
@ -9,7 +9,7 @@ terraform {

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/aws/fedora-coreos/kubernetes/workers.tf
+++ b/aws/fedora-coreos/kubernetes/workers.tf
@ -9,6 +9,7 @@ module "workers" {
  worker_count    = var.worker_count
  instance_type   = var.worker_type
  os_stream       = var.os_stream
+  arch            = var.arch
  disk_size       = var.disk_size
  spot_price      = var.worker_price
  target_groups   = var.worker_target_groups
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -18,3 +18,29 @@ data "aws_ami" "fedora-coreos" {
    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
+
+# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
+# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
+# and may be removed for any reason before then as well. Do not use.
+data "aws_ami" "fedora-coreos-arm" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["099663496933"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["fedora-coreos-*"]
+  }
+}
+
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -25,7 +25,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -39,12 +39,10 @@ systemd:
          --network host \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
          --volume /var/lib/calico:/var/lib/calico:ro \
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@ -70,6 +68,9 @@ systemd:
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
          %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
          --rotate-certificates \
@ -86,10 +87,11 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.19.3 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /var/lib/kubelet:/var/lib/kubelet:ro,z --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/aws/fedora-coreos/kubernetes/workers/variables.tf
+++ b/aws/fedora-coreos/kubernetes/workers/variables.tf
@ -36,8 +36,13 @@ variable "instance_type" {

 variable "os_stream" {
  type        = string
-  description = "Fedora CoreOs image stream for instances (e.g. stable, testing, next)"
+  description = "Fedora CoreOS image stream for instances (e.g. stable, testing, next)"
  default     = "stable"
+
+  validation {
+    condition     = contains(["stable", "testing", "next"], var.os_stream)
+    error_message = "The os_stream must be stable, testing, or next."
+  }
 }

 variable "disk_size" {
@ -108,3 +113,22 @@ variable "node_labels" {
  description = "List of initial node labels"
  default     = []
 }
+
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
--- a/aws/fedora-coreos/kubernetes/workers/versions.tf
+++ b/aws/fedora-coreos/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
    template = "~> 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/aws/fedora-coreos/kubernetes/workers/workers.tf
+++ b/aws/fedora-coreos/kubernetes/workers/workers.tf
@ -44,7 +44,7 @@ resource "aws_autoscaling_group" "workers" {

 # Worker template
 resource "aws_launch_configuration" "worker" {
-  image_id          = data.aws_ami.fedora-coreos.image_id
+  image_id          = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
  instance_type     = var.instance_type
  spot_price        = var.spot_price > 0 ? var.spot_price : null
  enable_monitoring = false
@ -86,6 +86,7 @@ data "template_file" "worker-config" {
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
    node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
  }
 }

--- a/aws/container-linux/kubernetes/LICENSE
+++ b/aws/container-linux/kubernetes/LICENSE
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,13 +11,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
+Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/flatcar-linux/aws/).

--- a/aws/container-linux/kubernetes/ami.tf
+++ b/aws/container-linux/kubernetes/ami.tf
@ -1,7 +1,7 @@
 locals {
  # Pick a Flatcar Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id = data.aws_ami.flatcar.image_id
+  ami_id  = data.aws_ami.flatcar.image_id
  channel = split("-", var.os_image)[1]
 }

--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=7988fb7159cb81e2d080b365b147fe90542fd258"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4edd79dd0295e6ffa4c8ed04fd5914d5cb3f1b7c"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers          = aws_route53_record.etcds.*.fqdn
-  asset_dir             = var.asset_dir
  networking            = var.networking
  network_mtu           = var.network_mtu
  pod_cidr              = var.pod_cidr
--- a/aws/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/aws/flatcar-linux/kubernetes/cl/controller.yaml
@ -0,0 +1,213 @@
+---
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/etcd-io/etcd
+        Requires=docker.service
+        After=docker.service
+        [Service]
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        ExecStartPre=/usr/bin/docker run -d \
+          --name etcd \
+          --network host \
+          --env-file /etc/etcd/etcd.env \
+          --user 232:232 \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro \
+          --volume /var/lib/etcd:/var/lib/etcd:rw \
+          $${ETCD_IMAGE}
+        ExecStart=docker logs -f etcd
+        ExecStop=docker stop etcd
+        ExecStopPost=docker rm etcd
+        Restart=always
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet (System Container)
+        Requires=docker.service
+        After=docker.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --healthz-port=0 \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        Wants=docker.service
+        After=docker.service
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        ExecStart=/usr/bin/docker run \
+            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
+            -v /opt/bootstrap/assets:/assets:ro \
+            -v /opt/bootstrap/apply:/apply:ro \
+            --entrypoint=/apply \
+            $${KUBELET_IMAGE}
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /var/lib/etcd
+      filesystem: root
+      mode: 0700
+      overwrite: true
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /opt/bootstrap/layout
+      filesystem: root
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/pki
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          chmod -R 700 /var/lib/etcd
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests-networking
+    - path: /opt/bootstrap/apply
+      filesystem: root
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/etcd/etcd.env
+      filesystem: root
+      mode: 0644
+      contents:
+          inline: |
+            ETCD_NAME=${etcd_name}
+            ETCD_DATA_DIR=/var/lib/etcd
+            ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+            ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+            ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+            ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+            ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+            ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+            ETCD_STRICT_RECONFIG_CHECK=true
+            ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+            ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+            ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+            ETCD_CLIENT_CERT_AUTH=true
+            ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+            ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+            ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+            ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
--- a/aws/container-linux/kubernetes/network.tf
+++ b/aws/container-linux/kubernetes/network.tf
--- a/aws/container-linux/kubernetes/nlb.tf
+++ b/aws/container-linux/kubernetes/nlb.tf
@ -17,6 +17,7 @@ resource "aws_route53_record" "apiserver" {
 resource "aws_lb" "nlb" {
  name               = "${var.cluster_name}-nlb"
  load_balancer_type = "network"
+  ip_address_type    = "dualstack"
  internal           = false

  subnets = aws_subnet.public.*.id
--- a/aws/container-linux/kubernetes/outputs.tf
+++ b/aws/container-linux/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "worker_security_groups" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom load balancing
@ -52,3 +54,10 @@ output "worker_target_group_https" {
  value       = module.workers.target_group_https
 }

+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/aws/container-linux/kubernetes/security.tf
+++ b/aws/container-linux/kubernetes/security.tf
--- a/aws/container-linux/kubernetes/ssh.tf
+++ b/aws/container-linux/kubernetes/ssh.tf
--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -45,6 +45,11 @@ variable "os_image" {
  type        = string
  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
  default     = "flatcar-stable"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+  }
 }

 variable "disk_size" {
@ -149,12 +154,6 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
--- a/aws/container-linux/kubernetes/versions.tf
+++ b/aws/container-linux/kubernetes/versions.tf
@ -1,7 +1,7 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
    template = "~> 2.1"
@ -9,7 +9,7 @@ terraform {

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/aws/container-linux/kubernetes/workers.tf
+++ b/aws/container-linux/kubernetes/workers.tf
--- a/aws/container-linux/kubernetes/workers/ami.tf
+++ b/aws/container-linux/kubernetes/workers/ami.tf
@ -1,7 +1,7 @@
 locals {
  # Pick a Flatcar Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id = data.aws_ami.flatcar.image_id
+  ami_id  = data.aws_ami.flatcar.image_id
  channel = split("-", var.os_image)[1]
 }

--- a/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -0,0 +1,117 @@
+---
+systemd:
+  units:
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet
+        Requires=docker.service
+        After=docker.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        # Podman, rkt, or runc run container processes, whereas docker run
+        # is a client to a daemon and requires workarounds to use within a
+        # systemd unit. https://github.com/moby/moby/issues/6791
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --healthz-port=0 \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/node \
+          %{~ for label in split(",", node_labels) ~}
+          --node-labels=${label} \
+          %{~ endfor ~}
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --rotate-certificates \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=5
+        [Install]
+        WantedBy=multi-user.target
+    - name: delete-node.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Delete Kubernetes node on shutdown
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/true
+        ExecStop=/bin/bash -c '/usr/bin/docker run -v /var/lib/kubelet:/var/lib/kubelet:ro --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/workers/ingress.tf
+++ b/aws/container-linux/kubernetes/workers/ingress.tf
--- a/aws/container-linux/kubernetes/workers/outputs.tf
+++ b/aws/container-linux/kubernetes/workers/outputs.tf
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -38,6 +38,11 @@ variable "os_image" {
  type        = string
  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
  default     = "flatcar-stable"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+  }
 }

 variable "disk_size" {
--- a/aws/container-linux/kubernetes/workers/versions.tf
+++ b/aws/container-linux/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
    template = "~> 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
--- a/azure/container-linux/kubernetes/cl/controller.yaml
+++ b/azure/container-linux/kubernetes/cl/controller.yaml
@ -1,205 +0,0 @@
---
-systemd:
-  units:
-    - name: etcd-member.service
-      enabled: true
-      dropins:
-        - name: 40-etcd-cluster.conf
-          contents: |
-            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.12"
-            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
-            Environment="RKT_RUN_ARGS=--insecure-options=image"
-            Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
-            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
-            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
-            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
-            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
-            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
-            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
-            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
-            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
-            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
-            Environment="ETCD_CLIENT_CERT_AUTH=true"
-            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
-            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
-            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
-            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-        RequiredBy=etcd-member.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=10
-        [Install]
-        WantedBy=multi-user.target
-    - name: bootstrap.service
-      contents: |
-        [Unit]
-        Description=Kubernetes control plane
-        ConditionPathExists=!/opt/bootstrap/bootstrap.done
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        WorkingDirectory=/opt/bootstrap
-        ExecStart=/usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes/bootstrap-secrets \
-            --mount volume=config,target=/etc/kubernetes/secrets \
-            --volume assets,kind=host,source=/opt/bootstrap/assets \
-            --mount volume=assets,target=/assets \
-            --volume script,kind=host,source=/opt/bootstrap/apply \
-            --mount volume=script,target=/apply \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.19.3 \
-            --net=host \
-            --dns=host \
-            --exec=/apply
-        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  directories:
-    - path: /var/lib/etcd
-      filesystem: root
-      mode: 0700
-      overwrite: true
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /opt/bootstrap/layout
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
-          awk '/#####/ {filename=$2; next} {print > filename}' assets
-          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
-          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
-          chown -R etcd:etcd /etc/ssl/etcd
-          chmod -R 500 /etc/ssl/etcd
-          chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          mkdir -p /etc/kubernetes/manifests
-          mv static-manifests/* /etc/kubernetes/manifests/
-          mkdir -p /opt/bootstrap/assets
-          mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls manifests-networking
-    - path: /opt/bootstrap/apply
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
-          until kubectl version; do
-            echo "Waiting for static pod control plane"
-            sleep 5
-          done
-          until kubectl apply -f /assets/manifests -R; do
-             echo "Retry applying manifests"
-             sleep 5
-          done
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml
@ -1,140 +0,0 @@
---
-systemd:
-  units:
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/node \
-          %{~ for label in split(",", node_labels) ~}
-          --node-labels=${label} \
-          %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=5
-        [Install]
-        WantedBy=multi-user.target
-    - name: delete-node.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Waiting to delete Kubernetes node on shutdown
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/etc/kubernetes/delete-node
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-    - path: /etc/kubernetes/delete-node
-      filesystem: root
-      mode: 0744
-      contents:
-        inline: |
-          #!/bin/bash
-          set -e
-          exec /usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes \
-            --mount volume=config,target=/etc/kubernetes \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.19.3 \
-            --net=host \
-            --dns=host \
-            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=7988fb7159cb81e2d080b365b147fe90542fd258"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4edd79dd0295e6ffa4c8ed04fd5914d5cb3f1b7c"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
-  asset_dir    = var.asset_dir

  networking = var.networking

--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -8,28 +8,25 @@ systemd:
      contents: |
        [Unit]
        Description=etcd (System Container)
-        Documentation=https://github.com/coreos/etcd
+        Documentation=https://github.com/etcd-io/etcd
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        # https://github.com/opencontainers/runc/pull/1807
-        # Type=notify
-        # NotifyAccess=exec
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
        Type=exec
-        Restart=on-failure
-        RestartSec=10s
-        TimeoutStartSec=0
-        LimitNOFILE=40000
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
-        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
        ExecStart=/usr/bin/podman run --name etcd \
          --env-file /etc/etcd/etcd.env \
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.12
+          $${ETCD_IMAGE}
        ExecStop=/usr/bin/podman stop etcd
+        Restart=on-failure
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
    - name: docker.service
@ -54,7 +51,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -68,12 +65,10 @@ systemd:
          --network host \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
          --volume /var/lib/calico:/var/lib/calico:ro \
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@ -119,11 +114,11 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
+            --volume /etc/kubernetes/pki:/etc/kubernetes/pki:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.19.3
+            quay.io/poseidon/kubelet:v1.20.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -146,26 +141,26 @@ storage:
          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
          awk '/#####/ {filename=$2; next} {print > filename}' assets
          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mkdir -p /etc/kubernetes/pki
          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
          chown -R etcd:etcd /etc/ssl/etcd
          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
          mkdir -p /etc/kubernetes/manifests
          mv static-manifests/* /etc/kubernetes/manifests/
          mkdir -p /opt/bootstrap/assets
          mv manifests /opt/bootstrap/assets/manifests
          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
-          chcon -R -u system_u -t container_file_t /etc/kubernetes/bootstrap-secrets
+          chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
        inline: |
          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
          until kubectl version; do
            echo "Waiting for static pod control plane"
            sleep 5
@ -201,8 +196,6 @@ storage:
      mode: 0644
      contents:
        inline: |
-          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
-          NOTIFY_SOCKET=/run/systemd/notify
          ETCD_NAME=${etcd_name}
          ETCD_DATA_DIR=/var/lib/etcd
          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
--- a/azure/fedora-coreos/kubernetes/outputs.tf
+++ b/azure/fedora-coreos/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "security_group_id" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom firewalling
@ -57,3 +59,11 @@ output "backend_address_pool_id" {
  description = "ID of the worker backend address pool"
  value       = azurerm_lb_backend_address_pool.worker.id
 }
+
+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/azure/fedora-coreos/kubernetes/variables.tf
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@ -129,12 +129,6 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -1,7 +1,7 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    azurerm  = "~> 2.8"
    template = "~> 2.1"
@ -9,7 +9,7 @@ terraform {

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -24,7 +24,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -38,12 +38,10 @@ systemd:
          --network host \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
          --volume /var/lib/calico:/var/lib/calico:ro \
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@ -85,10 +83,11 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.19.3 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /var/lib/kubelet:/var/lib/kubelet:ro,z --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/azure/fedora-coreos/kubernetes/workers/versions.tf
+++ b/azure/fedora-coreos/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    azurerm  = "~> 2.8"
    template = "~> 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/azure/container-linux/kubernetes/LICENSE
+++ b/azure/container-linux/kubernetes/LICENSE
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,13 +11,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

-Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/cl/azure/).
+Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/flatcar-linux/azure/).

--- a/azure/container-linux/kubernetes/bootstrap.tf
+++ b/azure/container-linux/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=7988fb7159cb81e2d080b365b147fe90542fd258"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4edd79dd0295e6ffa4c8ed04fd5914d5cb3f1b7c"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
-  asset_dir    = var.asset_dir

  networking = var.networking

--- a/azure/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/azure/flatcar-linux/kubernetes/cl/controller.yaml
@ -0,0 +1,213 @@
+---
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/etcd-io/etcd
+        Requires=docker.service
+        After=docker.service
+        [Service]
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        ExecStartPre=/usr/bin/docker run -d \
+          --name etcd \
+          --network host \
+          --env-file /etc/etcd/etcd.env \
+          --user 232:232 \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro \
+          --volume /var/lib/etcd:/var/lib/etcd:rw \
+          $${ETCD_IMAGE}
+        ExecStart=docker logs -f etcd
+        ExecStop=docker stop etcd
+        ExecStopPost=docker rm etcd
+        Restart=always
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet (System Container)
+        Requires=docker.service
+        After=docker.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --healthz-port=0 \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        Wants=docker.service
+        After=docker.service
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        ExecStart=/usr/bin/docker run \
+            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
+            -v /opt/bootstrap/assets:/assets:ro \
+            -v /opt/bootstrap/apply:/apply:ro \
+            --entrypoint=/apply \
+            $${KUBELET_IMAGE}
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /var/lib/etcd
+      filesystem: root
+      mode: 0700
+      overwrite: true
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /opt/bootstrap/layout
+      filesystem: root
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/pki
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          chmod -R 700 /var/lib/etcd
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests-networking
+    - path: /opt/bootstrap/apply
+      filesystem: root
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/etcd/etcd.env
+      filesystem: root
+      mode: 0644
+      contents:
+          inline: |
+            ETCD_NAME=${etcd_name}
+            ETCD_DATA_DIR=/var/lib/etcd
+            ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+            ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+            ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+            ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+            ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+            ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+            ETCD_STRICT_RECONFIG_CHECK=true
+            ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+            ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+            ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+            ETCD_CLIENT_CERT_AUTH=true
+            ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+            ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+            ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+            ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
--- a/azure/container-linux/kubernetes/lb.tf
+++ b/azure/container-linux/kubernetes/lb.tf
--- a/azure/container-linux/kubernetes/network.tf
+++ b/azure/container-linux/kubernetes/network.tf
--- a/azure/container-linux/kubernetes/outputs.tf
+++ b/azure/container-linux/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "security_group_id" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom firewalling
@ -57,3 +59,11 @@ output "backend_address_pool_id" {
  description = "ID of the worker backend address pool"
  value       = azurerm_lb_backend_address_pool.worker.id
 }
+
+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/azure/container-linux/kubernetes/security.tf
+++ b/azure/container-linux/kubernetes/security.tf
--- a/azure/container-linux/kubernetes/ssh.tf
+++ b/azure/container-linux/kubernetes/ssh.tf
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -50,6 +50,11 @@ variable "os_image" {
  type        = string
  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
  default     = "flatcar-stable"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+  }
 }

 variable "disk_size" {
@ -130,12 +135,6 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/azure/container-linux/kubernetes/versions.tf
+++ b/azure/container-linux/kubernetes/versions.tf
@ -1,7 +1,7 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    azurerm  = "~> 2.8"
    template = "~> 2.1"
@ -9,7 +9,7 @@ terraform {

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/azure/container-linux/kubernetes/workers.tf
+++ b/azure/container-linux/kubernetes/workers.tf
--- a/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -0,0 +1,117 @@
+---
+systemd:
+  units:
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet
+        Requires=docker.service
+        After=docker.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        # Podman, rkt, or runc run container processes, whereas docker run
+        # is a client to a daemon and requires workarounds to use within a
+        # systemd unit. https://github.com/moby/moby/issues/6791
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --healthz-port=0 \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/node \
+          %{~ for label in split(",", node_labels) ~}
+          --node-labels=${label} \
+          %{~ endfor ~}
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --rotate-certificates \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=5
+        [Install]
+        WantedBy=multi-user.target
+    - name: delete-node.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Delete Kubernetes node on shutdown
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/true
+        ExecStop=/bin/bash -c '/usr/bin/docker run -v /var/lib/kubelet:/var/lib/kubelet:ro --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@ -48,6 +48,11 @@ variable "os_image" {
  type        = string
  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
  default     = "flatcar-stable"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+  }
 }

 variable "priority" {
--- a/azure/container-linux/kubernetes/workers/versions.tf
+++ b/azure/container-linux/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    azurerm  = "~> 2.8"
    template = "~> 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }
  }
 }
--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml
@ -1,221 +0,0 @@
---
-systemd:
-  units:
-    - name: etcd-member.service
-      enabled: true
-      dropins:
-        - name: 40-etcd-cluster.conf
-          contents: |
-            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.12"
-            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
-            Environment="RKT_RUN_ARGS=--insecure-options=image"
-            Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
-            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
-            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
-            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
-            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
-            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
-            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
-            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
-            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
-            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
-            Environment="ETCD_CLIENT_CERT_AUTH=true"
-            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
-            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
-            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
-            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: kubelet.path
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Watch for kubeconfig
-        [Path]
-        PathExists=/etc/kubernetes/kubeconfig
-        [Install]
-        WantedBy=multi-user.target
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-        RequiredBy=etcd-member.service
-    - name: kubelet.service
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume etc-iscsi,kind=host,source=/etc/iscsi \
-          --mount volume=etc-iscsi,target=/etc/iscsi \
-          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
-          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --hostname-override=${domain_name} \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=10
-        [Install]
-        WantedBy=multi-user.target
-    - name: bootstrap.service
-      contents: |
-        [Unit]
-        Description=Kubernetes control plane
-        ConditionPathExists=!/opt/bootstrap/bootstrap.done
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        WorkingDirectory=/opt/bootstrap
-        ExecStart=/usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes/bootstrap-secrets \
-            --mount volume=config,target=/etc/kubernetes/secrets \
-            --volume assets,kind=host,source=/opt/bootstrap/assets \
-            --mount volume=assets,target=/assets \
-            --volume script,kind=host,source=/opt/bootstrap/apply \
-            --mount volume=script,target=/apply \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.19.3 \
-            --net=host \
-            --dns=host \
-            --exec=/apply
-        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  directories:
-    - path: /var/lib/etcd
-      filesystem: root
-      mode: 0700
-      overwrite: true
-    - path: /etc/kubernetes
-      filesystem: root
-      mode: 0755
-  files:
-    - path: /etc/hostname
-      filesystem: root
-      mode: 0644
-      contents:
-        inline:
-          ${domain_name}
-    - path: /opt/bootstrap/layout
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
-          awk '/#####/ {filename=$2; next} {print > filename}' assets
-          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
-          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
-          chown -R etcd:etcd /etc/ssl/etcd
-          chmod -R 500 /etc/ssl/etcd
-          chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          mkdir -p /etc/kubernetes/manifests
-          mv static-manifests/* /etc/kubernetes/manifests/
-          mkdir -p /opt/bootstrap/assets
-          mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls manifests-networking
-    - path: /opt/bootstrap/apply
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
-          until kubectl version; do
-            echo "Waiting for static pod control plane"
-            sleep 5
-          done
-          until kubectl apply -f /assets/manifests -R; do
-             echo "Retry applying manifests"
-             sleep 5
-          done
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - ${ssh_authorized_key}
--- a/bare-metal/container-linux/kubernetes/outputs.tf
+++ b/bare-metal/container-linux/kubernetes/outputs.tf
@ -1,4 +0,0 @@
-output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
-}
-
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=7988fb7159cb81e2d080b365b147fe90542fd258"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4edd79dd0295e6ffa4c8ed04fd5914d5cb3f1b7c"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
  etcd_servers                    = var.controllers.*.domain
-  asset_dir                       = var.asset_dir
  networking                      = var.networking
  network_mtu                     = var.network_mtu
  network_ip_autodetection_method = var.network_ip_autodetection_method
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -8,28 +8,25 @@ systemd:
      contents: |
        [Unit]
        Description=etcd (System Container)
-        Documentation=https://github.com/coreos/etcd
+        Documentation=https://github.com/etcd-io/etcd
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        # https://github.com/opencontainers/runc/pull/1807
-        # Type=notify
-        # NotifyAccess=exec
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
        Type=exec
-        Restart=on-failure
-        RestartSec=10s
-        TimeoutStartSec=0
-        LimitNOFILE=40000
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
-        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
        ExecStart=/usr/bin/podman run --name etcd \
          --env-file /etc/etcd/etcd.env \
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.12
+          $${ETCD_IMAGE}
        ExecStop=/usr/bin/podman stop etcd
+        Restart=on-failure
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
    - name: docker.service
@ -53,7 +50,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -67,20 +64,16 @@ systemd:
          --network host \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
          --volume /var/lib/calico:/var/lib/calico:ro \
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          --volume /etc/iscsi:/etc/iscsi \
-          --volume /sbin/iscsiadm:/sbin/iscsiadm \
          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -130,11 +123,11 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
+            --volume /etc/kubernetes/pki:/etc/kubernetes/pki:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.19.3
+            quay.io/poseidon/kubelet:v1.20.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -157,26 +150,26 @@ storage:
          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
          awk '/#####/ {filename=$2; next} {print > filename}' assets
          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mkdir -p /etc/kubernetes/pki
          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
          chown -R etcd:etcd /etc/ssl/etcd
          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
          mkdir -p /etc/kubernetes/manifests
          mv static-manifests/* /etc/kubernetes/manifests/
          mkdir -p /opt/bootstrap/assets
          mv manifests /opt/bootstrap/assets/manifests
          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
-          chcon -R -u system_u -t container_file_t /etc/kubernetes/bootstrap-secrets
+          chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
        inline: |
          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
          until kubectl version; do
            echo "Waiting for static pod control plane"
            sleep 5
@ -212,8 +205,6 @@ storage:
      mode: 0644
      contents:
        inline: |
-          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
-          NOTIFY_SOCKET=/run/systemd/notify
          ETCD_NAME=${etcd_name}
          ETCD_DATA_DIR=/var/lib/etcd
          ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -23,7 +23,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -37,20 +37,16 @@ systemd:
          --network host \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
          --volume /var/lib/calico:/var/lib/calico:ro \
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          --volume /etc/iscsi:/etc/iscsi \
-          --volume /sbin/iscsiadm:/sbin/iscsiadm \
          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
--- a/bare-metal/fedora-coreos/kubernetes/outputs.tf
+++ b/bare-metal/fedora-coreos/kubernetes/outputs.tf
@ -1,4 +1,12 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
+}
+
+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
 }

--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -1,26 +1,32 @@
 locals {
  remote_kernel = "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-live-kernel-x86_64"
-  remote_initrd = "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-live-initramfs.x86_64.img"
+  remote_initrd = [
+    "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-live-initramfs.x86_64.img",
+    "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-live-rootfs.x86_64.img"
+  ]
+
  remote_args = [
    "ip=dhcp",
    "rd.neednet=1",
-    "initrd=fedora-coreos-${var.os_version}-live-initramfs.x86_64.img",
-    "coreos.inst.image_url=https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-metal.x86_64.raw.xz",
-    "coreos.inst.ignition_url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.inst.install_dev=${var.install_disk}",
+    "coreos.inst.ignition_url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
+    "coreos.inst.image_url=https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-metal.x86_64.raw.xz",
    "console=tty0",
    "console=ttyS0",
  ]

  cached_kernel = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-kernel-x86_64"
-  cached_initrd = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-initramfs.x86_64.img"
+  cached_initrd = [
+    "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-initramfs.x86_64.img",
+    "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-rootfs.x86_64.img"
+  ]
+
  cached_args = [
    "ip=dhcp",
    "rd.neednet=1",
-    "initrd=fedora-coreos-${var.os_version}-live-initramfs.x86_64.img",
-    "coreos.inst.image_url=${var.matchbox_http_endpoint}/assets/fedora-coreos/fedora-coreos-${var.os_version}-metal.x86_64.raw.xz",
-    "coreos.inst.ignition_url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.inst.install_dev=${var.install_disk}",
+    "coreos.inst.ignition_url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
+    "coreos.inst.image_url=${var.matchbox_http_endpoint}/assets/fedora-coreos/fedora-coreos-${var.os_version}-metal.x86_64.raw.xz",
    "console=tty0",
    "console=ttyS0",
  ]
@ -37,9 +43,7 @@ resource "matchbox_profile" "controllers" {
  name  = format("%s-controller-%s", var.cluster_name, var.controllers.*.name[count.index])

  kernel = local.kernel
-  initrd = [
-    local.initrd
-  ]
+  initrd = local.initrd
  args = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.controller-ignitions.*.rendered[count.index]
@ -73,9 +77,7 @@ resource "matchbox_profile" "workers" {
  name  = format("%s-worker-%s", var.cluster_name, var.workers.*.name[count.index])

  kernel = local.kernel
-  initrd = [
-    local.initrd
-  ]
+  initrd = local.initrd
  args = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.worker-ignitions.*.rendered[count.index]
--- a/bare-metal/fedora-coreos/kubernetes/variables.tf
+++ b/bare-metal/fedora-coreos/kubernetes/variables.tf
@ -12,8 +12,13 @@ variable "matchbox_http_endpoint" {

 variable "os_stream" {
  type        = string
-  description = "Fedora CoreOS release stream (e.g. testing, stable)"
+  description = "Fedora CoreOS release stream (e.g. stable, testing, next)"
  default     = "stable"
+
+  validation {
+    condition     = contains(["stable", "testing", "next"], var.os_stream)
+    error_message = "The os_stream must be stable, testing, or next."
+  }
 }

 variable "os_version" {
@ -80,12 +85,6 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
  description = "Choice of networking provider (flannel or calico)"
--- a/bare-metal/fedora-coreos/kubernetes/versions.tf
+++ b/bare-metal/fedora-coreos/kubernetes/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    template = "~> 2.1"
    null     = "~> 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }

    matchbox = {
--- a/bare-metal/container-linux/kubernetes/LICENSE
+++ b/bare-metal/container-linux/kubernetes/LICENSE
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,13 +11,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.19.3 (upstream)
+* Kubernetes v1.20.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

-Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/cl/bare-metal/).
+Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/flatcar-linux/bare-metal/).

--- a/bare-metal/container-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/container-linux/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=7988fb7159cb81e2d080b365b147fe90542fd258"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4edd79dd0295e6ffa4c8ed04fd5914d5cb3f1b7c"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
  etcd_servers                    = var.controllers.*.domain
-  asset_dir                       = var.asset_dir
  networking                      = var.networking
  network_mtu                     = var.network_mtu
  network_ip_autodetection_method = var.network_ip_autodetection_method
--- a/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
@ -0,0 +1,225 @@
+---
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/etcd-io/etcd
+        Requires=docker.service
+        After=docker.service
+        [Service]
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        ExecStartPre=/usr/bin/docker run -d \
+          --name etcd \
+          --network host \
+          --env-file /etc/etcd/etcd.env \
+          --user 232:232 \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro \
+          --volume /var/lib/etcd:/var/lib/etcd:rw \
+          $${ETCD_IMAGE}
+        ExecStart=docker logs -f etcd
+        ExecStop=docker stop etcd
+        ExecStopPost=docker rm etcd
+        Restart=always
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: kubelet.path
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Watch for kubeconfig
+        [Path]
+        PathExists=/etc/kubernetes/kubeconfig
+        [Install]
+        WantedBy=multi-user.target
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      contents: |
+        [Unit]
+        Description=Kubelet (System Container)
+        Requires=docker.service
+        After=docker.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --healthz-port=0 \
+          --hostname-override=${domain_name} \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        Wants=docker.service
+        After=docker.service
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
+        ExecStart=/usr/bin/docker run \
+            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
+            -v /opt/bootstrap/assets:/assets:ro \
+            -v /opt/bootstrap/apply:/apply:ro \
+            --entrypoint=/apply \
+            $${KUBELET_IMAGE}
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /var/lib/etcd
+      filesystem: root
+      mode: 0700
+      overwrite: true
+    - path: /etc/kubernetes
+      filesystem: root
+      mode: 0755
+  files:
+    - path: /etc/hostname
+      filesystem: root
+      mode: 0644
+      contents:
+        inline:
+          ${domain_name}
+    - path: /opt/bootstrap/layout
+      filesystem: root
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/pki
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          chmod -R 700 /var/lib/etcd
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests-networking
+    - path: /opt/bootstrap/apply
+      filesystem: root
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/etcd/etcd.env
+      filesystem: root
+      mode: 0644
+      contents:
+          inline: |
+            ETCD_NAME=${etcd_name}
+            ETCD_DATA_DIR=/var/lib/etcd
+            ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379
+            ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380
+            ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+            ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+            ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+            ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+            ETCD_STRICT_RECONFIG_CHECK=true
+            ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+            ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+            ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+            ETCD_CLIENT_CERT_AUTH=true
+            ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+            ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+            ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+            ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - ${ssh_authorized_key}
--- a/bare-metal/container-linux/kubernetes/cl/install.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/install.yaml
@ -31,7 +31,7 @@ storage:
        inline: |
          #!/bin/bash -ex
          curl --retry 10 "${ignition_endpoint}?{{.request.raw_query}}&os=installed" -o ignition.json
-          ${os_flavor}-install \
+          flatcar-install \
            -d ${install_disk} \
            -C ${os_channel} \
            -V ${os_version} \
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml
@ -31,9 +31,11 @@ systemd:
      contents: |
        [Unit]
        Description=Kubelet
+        Requires=docker.service
+        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.0
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -41,43 +43,27 @@ systemd:
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume etc-iscsi,kind=host,source=/etc/iscsi \
-          --mount volume=etc-iscsi,target=/etc/iscsi \
-          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
-          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          $${KUBELET_IMAGE} -- \
+        # Podman, rkt, or runc run container processes, whereas docker run
+        # is a client to a daemon and requires workarounds to use within a
+        # systemd unit. https://github.com/moby/moby/issues/6791
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -102,7 +88,9 @@ systemd:
          --read-only-port=0 \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
        Restart=always
        RestartSec=5
        [Install]
--- a/bare-metal/container-linux/kubernetes/groups.tf
+++ b/bare-metal/container-linux/kubernetes/groups.tf
@ -3,8 +3,8 @@ resource "matchbox_group" "install" {

  name = format("install-%s", concat(var.controllers.*.name, var.workers.*.name)[count.index])

-  # pick one of 4 Matchbox profiles (Container Linux or Flatcar, cached or non-cached)
-  profile = local.flavor == "flatcar" ? var.cached_install ? matchbox_profile.cached-flatcar-linux-install.*.name[count.index] : matchbox_profile.flatcar-install.*.name[count.index] : var.cached_install ? matchbox_profile.cached-container-linux-install.*.name[count.index] : matchbox_profile.container-linux-install.*.name[count.index]
+  # pick Matchbox profile (Flatcar upstream or Matchbox image cache)
+  profile = var.cached_install ? matchbox_profile.cached-flatcar-install.*.name[count.index] : matchbox_profile.flatcar-install.*.name[count.index]

  selector = {
    mac = concat(var.controllers.*.mac, var.workers.*.mac)[count.index]
--- a/bare-metal/flatcar-linux/kubernetes/outputs.tf
+++ b/bare-metal/flatcar-linux/kubernetes/outputs.tf
@ -0,0 +1,12 @@
+output "kubeconfig-admin" {
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
+}
+
+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -1,91 +1,8 @@
 locals {
-  # coreos-stable -> coreos flavor, stable channel
-  # flatcar-stable -> flatcar flavor, stable channel
-  flavor  = split("-", var.os_channel)[0]
+  # flatcar-stable -> stable channel
  channel = split("-", var.os_channel)[1]
 }

-// Container Linux Install profile (from release.core-os.net)
-resource "matchbox_profile" "container-linux-install" {
-  count = length(var.controllers) + length(var.workers)
-  name  = format("%s-container-linux-install-%s", var.cluster_name, concat(var.controllers.*.name, var.workers.*.name)[count.index])
-
-  kernel = "${var.download_protocol}://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe.vmlinuz"
-
-  initrd = [
-    "${var.download_protocol}://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe_image.cpio.gz",
-  ]
-
-  args = flatten([
-    "initrd=coreos_production_pxe_image.cpio.gz",
-    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
-    "coreos.first_boot=yes",
-    "console=tty0",
-    "console=ttyS0",
-    var.kernel_args,
-  ])
-
-  container_linux_config = data.template_file.container-linux-install-configs.*.rendered[count.index]
-}
-
-data "template_file" "container-linux-install-configs" {
-  count = length(var.controllers) + length(var.workers)
-
-  template = file("${path.module}/cl/install.yaml")
-
-  vars = {
-    os_flavor          = local.flavor
-    os_channel         = local.channel
-    os_version         = var.os_version
-    ignition_endpoint  = format("%s/ignition", var.matchbox_http_endpoint)
-    install_disk       = var.install_disk
-    ssh_authorized_key = var.ssh_authorized_key
-    # only cached-container-linux profile adds -b baseurl
-    baseurl_flag = ""
-  }
-}
-
-// Container Linux Install profile (from matchbox /assets cache)
-// Note: Admin must have downloaded os_version into matchbox assets/coreos.
-resource "matchbox_profile" "cached-container-linux-install" {
-  count = length(var.controllers) + length(var.workers)
-  name  = format("%s-cached-container-linux-install-%s", var.cluster_name, concat(var.controllers.*.name, var.workers.*.name)[count.index])
-
-  kernel = "/assets/coreos/${var.os_version}/coreos_production_pxe.vmlinuz"
-
-  initrd = [
-    "/assets/coreos/${var.os_version}/coreos_production_pxe_image.cpio.gz",
-  ]
-
-  args = flatten([
-    "initrd=coreos_production_pxe_image.cpio.gz",
-    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
-    "coreos.first_boot=yes",
-    "console=tty0",
-    "console=ttyS0",
-    var.kernel_args,
-  ])
-
-  container_linux_config = data.template_file.cached-container-linux-install-configs.*.rendered[count.index]
-}
-
-data "template_file" "cached-container-linux-install-configs" {
-  count = length(var.controllers) + length(var.workers)
-
-  template = file("${path.module}/cl/install.yaml")
-
-  vars = {
-    os_flavor          = local.flavor
-    os_channel         = local.channel
-    os_version         = var.os_version
-    ignition_endpoint  = format("%s/ignition", var.matchbox_http_endpoint)
-    install_disk       = var.install_disk
-    ssh_authorized_key = var.ssh_authorized_key
-    # profile uses -b baseurl to install from matchbox cache
-    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/${local.flavor}"
-  }
-}
-
 // Flatcar Linux install profile (from release.flatcar-linux.net)
 resource "matchbox_profile" "flatcar-install" {
  count = length(var.controllers) + length(var.workers)
@ -106,12 +23,12 @@ resource "matchbox_profile" "flatcar-install" {
    var.kernel_args,
  ])

-  container_linux_config = data.template_file.container-linux-install-configs.*.rendered[count.index]
+  container_linux_config = data.template_file.install-configs.*.rendered[count.index]
 }

 // Flatcar Linux Install profile (from matchbox /assets cache)
 // Note: Admin must have downloaded os_version into matchbox assets/flatcar.
-resource "matchbox_profile" "cached-flatcar-linux-install" {
+resource "matchbox_profile" "cached-flatcar-install" {
  count = length(var.controllers) + length(var.workers)
  name  = format("%s-cached-flatcar-linux-install-%s", var.cluster_name, concat(var.controllers.*.name, var.workers.*.name)[count.index])

@ -130,9 +47,42 @@ resource "matchbox_profile" "cached-flatcar-linux-install" {
    var.kernel_args,
  ])

-  container_linux_config = data.template_file.cached-container-linux-install-configs.*.rendered[count.index]
+  container_linux_config = data.template_file.cached-install-configs.*.rendered[count.index]
 }

+data "template_file" "install-configs" {
+  count = length(var.controllers) + length(var.workers)
+
+  template = file("${path.module}/cl/install.yaml")
+
+  vars = {
+    os_channel         = local.channel
+    os_version         = var.os_version
+    ignition_endpoint  = format("%s/ignition", var.matchbox_http_endpoint)
+    install_disk       = var.install_disk
+    ssh_authorized_key = var.ssh_authorized_key
+    # only cached profile adds -b baseurl
+    baseurl_flag = ""
+  }
+}
+
+data "template_file" "cached-install-configs" {
+  count = length(var.controllers) + length(var.workers)
+
+  template = file("${path.module}/cl/install.yaml")
+
+  vars = {
+    os_channel         = local.channel
+    os_version         = var.os_version
+    ignition_endpoint  = format("%s/ignition", var.matchbox_http_endpoint)
+    install_disk       = var.install_disk
+    ssh_authorized_key = var.ssh_authorized_key
+    # profile uses -b baseurl to install from matchbox cache
+    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/flatcar"
+  }
+}
+
+
 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
  count        = length(var.controllers)
--- a/bare-metal/container-linux/kubernetes/ssh.tf
+++ b/bare-metal/container-linux/kubernetes/ssh.tf
--- a/bare-metal/container-linux/kubernetes/variables.tf
+++ b/bare-metal/container-linux/kubernetes/variables.tf
@ -13,11 +13,16 @@ variable "matchbox_http_endpoint" {
 variable "os_channel" {
  type        = string
  description = "Channel for a Flatcar Linux (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_channel)
+    error_message = "The os_channel must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+  }
 }

 variable "os_version" {
  type        = string
-  description = "Version for a Container Linux derivative to PXE and install (e.g. 2079.5.1)"
+  description = "Version of Flatcar Linux to PXE and install (e.g. 2079.5.1)"
 }

 # machines
@ -122,14 +127,14 @@ variable "download_protocol" {

 variable "cached_install" {
  type        = bool
-  description = "Whether Container Linux should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the os_version into matchbox assets."
+  description = "Whether Flatcar Linux should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the os_version into matchbox assets."
  default     = false
 }

 variable "install_disk" {
  type        = string
  default     = "/dev/sda"
-  description = "Disk device to which the install profiles should install Container Linux (e.g. /dev/sda)"
+  description = "Disk device to which the install profiles should install Flatcar Linux (e.g. /dev/sda)"
 }

 variable "kernel_args" {
@ -152,12 +157,6 @@ variable "enable_aggregation" {

 # unofficial, undocumented, unsupported

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
--- a/bare-metal/container-linux/kubernetes/versions.tf
+++ b/bare-metal/container-linux/kubernetes/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12.26, < 0.14.0"
+  required_version = "~> 0.13.0"
  required_providers {
    template = "~> 2.1"
    null     = "~> 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6.1"
+      version = "~> 0.6"
    }

    matchbox = {
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml
@ -1,207 +0,0 @@
---
-systemd:
-  units:
-    - name: etcd-member.service
-      enabled: true
-      dropins:
-        - name: 40-etcd-cluster.conf
-          contents: |
-            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.12"
-            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
-            Environment="RKT_RUN_ARGS=--insecure-options=image"
-            Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
-            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
-            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
-            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
-            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
-            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
-            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
-            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
-            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
-            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
-            Environment="ETCD_CLIENT_CERT_AUTH=true"
-            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
-            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
-            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
-            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: kubelet.path
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Watch for kubeconfig
-        [Path]
-        PathExists=/etc/kubernetes/kubeconfig
-        [Install]
-        WantedBy=multi-user.target
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-        RequiredBy=etcd-member.service
-    - name: kubelet.service
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.19.3
-        EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=10
-        [Install]
-        WantedBy=multi-user.target
-    - name: bootstrap.service
-      contents: |
-        [Unit]
-        Description=Kubernetes control plane
-        ConditionPathExists=!/opt/bootstrap/bootstrap.done
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        WorkingDirectory=/opt/bootstrap
-        ExecStart=/usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes/bootstrap-secrets \
-            --mount volume=config,target=/etc/kubernetes/secrets \
-            --volume assets,kind=host,source=/opt/bootstrap/assets \
-            --mount volume=assets,target=/assets \
-            --volume script,kind=host,source=/opt/bootstrap/apply \
-            --mount volume=script,target=/apply \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.19.3 \
-            --net=host \
-            --dns=host \
-            --exec=/apply
-        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  directories:
-    - path: /var/lib/etcd
-      filesystem: root
-      mode: 0700
-      overwrite: true
-    - path: /etc/kubernetes
-      filesystem: root
-      mode: 0755
-  files:
-    - path: /opt/bootstrap/layout
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
-          awk '/#####/ {filename=$2; next} {print > filename}' assets
-          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
-          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
-          chown -R etcd:etcd /etc/ssl/etcd
-          chmod -R 500 /etc/ssl/etcd
-          chmod -R 700 /var/lib/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          mkdir -p /etc/kubernetes/manifests
-          mv static-manifests/* /etc/kubernetes/manifests/
-          mkdir -p /opt/bootstrap/assets
-          mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls manifests-networking
-    - path: /opt/bootstrap/apply
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
-          until kubectl version; do
-            echo "Waiting for static pod control plane"
-            sleep 5
-          done
-          until kubectl apply -f /assets/manifests -R; do
-             echo "Retry applying manifests"
-             sleep 5
-          done
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Dalton Hubble	c163fbbbcd	Update docs and README for release	2020-12-12 12:31:35 -08:00
Dalton Hubble	dc7be431e0	Remove iSCSI mounts from Kubelet * Remove Kubelet `/etc/iscsi` and `iscsiadm` host mounts that were added on bare-metal, since these no longer work on either Fedora CoreOS or Flatcar Linux with newer `iscsiadm` * These special mounts on bare-metal date back to #350 which added them to provide a way to use iSCSI in Kubernetes v1.10 * Today, storage should be handled by external CSI providers which handle different storage systems, which doesn't rely on Kubelet storage utils Close #907	2020-12-12 11:41:02 -08:00
Dalton Hubble	86e0f806b3	Revert "Add support for Terraform v0.14.x" This reverts commit `968febb050`.	2020-12-11 00:47:57 -08:00
Dalton Hubble	96172ad269	Update Grafana from v7.3.4 to v7.3.5 * https://github.com/grafana/grafana/releases/tag/v7.3.5	2020-12-11 00:24:43 -08:00
Dalton Hubble	3eb20a1f4b	Update recommended Terraform provider versions * Sync Terraform provider plugins with those used internally	2020-12-11 00:15:29 -08:00
Dalton Hubble	ee9ce3d0ab	Update Calico from v3.17.0 to v3.17.1 * https://github.com/projectcalico/calico/releases/tag/v3.17.1	2020-12-10 22:48:38 -08:00
Dalton Hubble	a8b8a9b454	Update Kubernetes from v1.20.0-rc.0 to v1.20.0 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1200	2020-12-08 18:28:13 -08:00
Dalton Hubble	968febb050	Add support for Terraform v0.14.x * Support Terraform v0.13.x and v0.14.x	2020-12-07 00:22:38 -08:00
Dalton Hubble	bee455f83a	Update Cilium from v1.9.0 to v1.9.1 * https://github.com/cilium/cilium/releases/tag/v1.9.1	2020-12-04 14:14:18 -08:00
Dalton Hubble	3e89ea1b4a	Promote Fedora CoreOS bare-metal to stable * Fedora CoreOS is a good choice for use on bare-metal	2020-12-04 14:02:55 -08:00
Dalton Hubble	e77dd6ecd4	Update Kubernetes from v1.19.4 to v1.20.0-rc.0 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1200-rc0	2020-12-03 16:01:28 -08:00
Dalton Hubble	4fd4a0f540	Move control plane static pod TLS assets to /etc/kubernetes/pki * Change control plane static pods to mount `/etc/kubernetes/pki`, instead of `/etc/kubernetes/bootstrap-secrets` to better reflect their purpose and match some loose conventions upstream * Place control plane and bootstrap TLS assets and kubeconfig's in `/etc/kubernetes/pki` * Mount to `/etc/kubernetes/pki` (rather than `/etc/kubernetes/secrets`) to match the host location (less surprise) Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/233	2020-12-02 23:26:42 -08:00
Dalton Hubble	804dfea0f9	Add kubeconfig's for kube-scheduler and kube-controller-manager * Generate TLS client certificates for `kube-scheduler` and `kube-controller-manager` with `system:kube-scheduler` and `system:kube-controller-manager` CNs * Template separate kubeconfigs for kube-scheduler and kube-controller manager (`scheduler.conf` and `controller-manager.conf`). Rename admin for clarity * Before v1.16.0, Typhoon scheduled a self-hosted control plane, which allowed the steady-state kube-scheduler and kube-controller-manager to use a scoped ServiceAccount. With a static pod control plane, separate CN TLS client certificates are the nearest equiv. * https://kubernetes.io/docs/setup/best-practices/certificates/ * Remove unused Kubelet certificate, TLS bootstrap is used instead	2020-12-01 22:02:15 -08:00
Dalton Hubble	8ba23f364c	Add TokenReview and TokenRequestProjection flags * Add kube-apiserver flags for TokenReview and TokenRequestProjection (beta, defaults on) to allow using Service Account Token Volume Projection to create and mount service account tokens tied to a Pod's lifecycle Rel: * https://github.com/poseidon/terraform-render-bootstrap/pull/231 * https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection	2020-12-01 20:02:33 -08:00
Dalton Hubble	f6025666eb	Update etcd from v3.4.12 to v3.4.14 * https://github.com/etcd-io/etcd/releases/tag/v3.4.14	2020-11-29 20:04:25 -08:00
Dalton Hubble	85eb502f19	Update Prometheus from v2.23.0-rc.0 to v2.23.0 * https://github.com/prometheus/prometheus/releases/tag/v2.23.0	2020-11-29 19:59:27 -08:00
Dalton Hubble	fa3184fb9c	Relax terraform-provider-ct version constraint * Allow terraform-provider-ct versions v0.6+ (e.g. v0.7.1) Before, only v0.6.x point updates were allowed * Update terraform-provider-ct to v0.7.1 in docs * READ the docs before updating terraform-provider-ct, as changing worker user-data is handled differently by different cloud platforms	2020-11-29 19:51:26 -08:00
Dalton Hubble	22565e57e0	Update kube-state-metrics from v2.0.0-alpha.2 to v2.0.0-alpha.3 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.3	2020-11-25 14:30:11 -08:00
Dalton Hubble	026e1f3648	Update Grafana from v7.3.3 to v7.3.4 * https://github.com/grafana/grafana/releases/tag/v7.3.4	2020-11-25 14:25:15 -08:00
Dalton Hubble	ae548ce213	Update Calico from v3.16.5 to v3.17.0 * Enable Calico MTU auto-detection * Remove [workaround](https://github.com/poseidon/typhoon/pull/724) to Calico cni-plugin [issue](https://github.com/projectcalico/cni-plugin/issues/874) Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/230	2020-11-25 14:22:58 -08:00
Dalton Hubble	e826b49648	Update Matchbox profile to use initramfs and rootfs images * Fedora CoreOS stable (after Oct 6) ships separate initramfs and rootfs images, used as initrd's * Update profiles to match the Matchbox examples, which have already switched to the new profile and to remove the unused kernel args * Requires Fedora CoreOS version which ships rootfs images (e.g. stable 32.20200923.3.0 or later) Rel: * https://github.com/coreos/fedora-coreos-tracker/issues/390#issuecomment-661986987 * `da0df01763 (diff-4541f7b7c174f6ae6270135942c1c65ed9e09ebe81239709f5a9fb34e858ddcf)` Supercedes https://github.com/poseidon/typhoon/pull/888	2020-11-25 14:13:39 -08:00
Dalton Hubble	fa8f68f50e	Fix Fedora CoreOS AWS AMI query in non-US regions * A `aws_ami` data source will fail a Terraform plan if no matching AMI is found, even if the AMI is not used. ARM64 images are only published to a few US regions, so the `aws_ami` data query could fail when creating Fedora CoreOS AWS clusters in non-US regions * Condition `aws_ami` on whether experimental arch `arm64` is chosen * Recent regression introduced in v1.19.4 https://github.com/poseidon/typhoon/pull/875 Closes https://github.com/poseidon/typhoon/issues/886	2020-11-25 11:32:05 -08:00
Dalton Hubble	ba8d972c76	Update Prometheus from v2.22.2 to v2.23.0-rc.0 * https://github.com/prometheus/prometheus/releases/tag/v2.23.0-rc.0	2020-11-24 10:54:42 -08:00
Dalton Hubble	c0347ca0c6	Set kubeconfig and asset_dist as sensitive * Mark `kubeconfig` and `asset_dist` as `sensitive` to prevent the Terraform CLI displaying these values, esp. for CI systems * In particular, external tools or tfvars style uses (not recommended) reportedly display all outputs and are improved by setting sensitive * For Terraform v0.14, outputs referencing sensitive fields must also be annotated as sensitive Closes https://github.com/poseidon/typhoon/issues/884	2020-11-23 11:41:55 -08:00
Dalton Hubble	9f94ab6bcc	Rerun terraform fmt for recent variables	2020-11-21 14:20:36 -08:00
Dalton Hubble	5e4f5de271	Enable Network Load Balancer (NLB) dualstack * NLB subnets assigned both IPv4 and IPv6 addresses * NLB DNS name has both A and AAAA records * NLB to target node traffic is IPv4 (no change), no change to security groups needed * Ingresses exposed through the recommended Nginx Ingress Controller addon will be accessible via IPv4 or IPv6. No change is needed to the app's CNAME to NLB record Related: https://aws.amazon.com/about-aws/whats-new/2020/11/network-load-balancer-supports-ipv6/	2020-11-21 14:16:24 -08:00
Dalton Hubble	be28495d79	Update Prometheus from v2.22.1 to v2.22.2 * https://github.com/prometheus/prometheus/releases/tag/v2.22.2	2020-11-19 21:50:48 -08:00
Dalton Hubble	f1356fec24	Update Grafana from v7.3.2 to v7.3.3 * https://github.com/grafana/grafana/releases/tag/v7.3.3	2020-11-19 21:49:11 -08:00
Dalton Hubble	cc00afa4e1	Add Terraform v0.13 input variable validations * Support for migrating from Terraform v0.12.x to v0.13.x was added in v1.18.8 * Require Terraform v0.13+. Drop support for Terraform v0.12	2020-11-17 12:02:34 -08:00
Dalton Hubble	5c3b5a20de	Update recommended Terraform provider versions * Sync Terraform provider plugins with those used internally	2020-11-14 13:32:04 -08:00
Dalton Hubble	f5a83667e8	Update Grafana from v7.3.1 to v7.3.2 * https://github.com/grafana/grafana/releases/tag/v7.3.2	2020-11-14 13:30:30 -08:00
Dalton Hubble	a911367c2e	Update nginx-ingress from v0.41.0 to v0.41.2 * https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.41.2	2020-11-14 13:27:06 -08:00
Dalton Hubble	f884de847e	Discard Prometheus etcd gRPC failure alert * Kubernetes watch expiry is not a gRPC code we care about * Background: This rule is typically removed, but was added back in	2020-11-14 13:17:56 -08:00
Dalton Hubble	1b3a0f6ebc	Add experimental Fedora CoreOS arm64 support on AWS * Add experimental `arch` variable to Fedora CoreOS AWS, accepting amd64 (default) or arm64 to support native arm64/aarch64 clusters or mixed/hybrid clusters with a worker pool of arm64 workers * Add `daemonset_tolerations` variable to cluster module (experimental) * Add `node_taints` variable to workers module * Requires flannel CNI and experimental Poseidon-built arm64 Fedora CoreOS AMIs (published to us-east-1, us-east-2, and us-west-1) WARN: * Our AMIs are experimental, may be removed at any time, and will be removed when Fedora CoreOS publishes official arm64 AMIs. Do NOT use in production Related: * https://github.com/poseidon/typhoon/pull/682	2020-11-14 13:09:24 -08:00
Dalton Hubble	1113a22f61	Update Kubernetes from v1.19.3 to v1.19.4 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1194	2020-11-11 22:56:27 -08:00
Dalton Hubble	152c7d86bd	Change bootstrap.service container from rkt to docker * Use docker to run `bootstrap.service` container * Background https://github.com/poseidon/typhoon/pull/855	2020-11-11 22:26:05 -08:00
Dalton Hubble	79deb8a967	Update Cilium from v1.9.0-rc3 to v1.9.0 * https://github.com/cilium/cilium/releases/tag/v1.9.0	2020-11-10 23:42:41 -08:00
Dalton Hubble	f412f0d9f2	Update Calico from v3.16.4 to v3.16.5 * https://github.com/projectcalico/calico/releases/tag/v3.16.5	2020-11-10 22:58:19 -08:00
Phil Sautter	eca6c4a1a1	Fix broken flatcar linux documentation links (#870 ) * Fix old documentation links	2020-11-10 18:30:30 -08:00
Dalton Hubble	133d325013	Update nginx-ingress from v0.40.2 to v0.41.0 * https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.41.0	2020-11-08 14:34:52 -08:00
Dalton Hubble	4b05c0180e	Update Grafana from v7.3.0 to v7.3.1 * https://github.com/grafana/grafana/releases/tag/v7.3.1	2020-11-08 14:13:39 -08:00
Dalton Hubble	f49ab3a6ee	Update Prometheus from v2.22.0 to v2.22.1 * https://github.com/prometheus/prometheus/releases/tag/v2.22.1	2020-11-08 14:12:24 -08:00
Dalton Hubble	0eef16b274	Improve and tidy Fedora CoreOS etcd-member.service * Allow a snippet with a systemd dropin to set an alternate image via `ETCD_IMAGE`, for consistency across Fedora CoreOS and Flatcar Linux * Drop comments about integrating system containers with systemd-notify	2020-11-08 11:49:56 -08:00
Dalton Hubble	ad1f59ce91	Change Flatcar etcd-member.service container from rkt to docker * Use docker to run the `etcd-member.service` container * Use env-file `/etc/etcd/etcd.env` like podman on FCOS * Background: https://github.com/poseidon/typhoon/pull/855	2020-11-03 16:42:18 -08:00
Dalton Hubble	82e5ac3e7c	Update Cilium from v1.8.5 to v1.9.0-rc3 * https://github.com/poseidon/terraform-render-bootstrap/pull/224	2020-11-03 10:29:07 -08:00
Dalton Hubble	a8f7880511	Update Cilium from v1.8.4 to v1.8.5 * https://github.com/cilium/cilium/releases/tag/v1.8.5	2020-10-29 00:50:18 -07:00
Dalton Hubble	cda5b93b09	Update kube-state-metrics from v2.0.0-alpha.1 to v2.0.0-alpha.2 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.2	2020-10-28 18:49:40 -07:00
Dalton Hubble	3e9f5f34de	Update Grafana from v7.2.2 to v7.3.0 * https://github.com/grafana/grafana/releases/tag/v7.3.0	2020-10-28 17:46:26 -07:00
Dalton Hubble	893d139590	Update Calico from v3.16.3 to v3.16.4 * https://github.com/projectcalico/calico/releases/tag/v3.16.4	2020-10-26 00:50:40 -07:00
Dalton Hubble	fc62e51b2a	Update Grafana from v7.2.1 to v7.2.2 * https://github.com/grafana/grafana/releases/tag/v7.2.2	2020-10-22 00:14:04 -07:00
Dalton Hubble	e5ba3329eb	Remove bare-metal CoreOS Container Linux profiles * Remove Matchbox profiles for CoreOS Container Linux * Simplify the remaining Flatcat Linux profiles	2020-10-21 00:25:10 -07:00
Dalton Hubble	7c3f3ab6d0	Rename container-linux modules to flatcar-linux * CoreOS Container Linux was deprecated in v1.18.3 * Continue transitioning docs and modules from supporting both CoreOS and Flatcar "variants" of Container Linux to now supporting Flatcar Linux and equivalents Action Required: Update the Flatcar Linux modules `source` to replace `s/container-linux/flatcar-linux`. See docs for examples	2020-10-20 22:47:19 -07:00
Dalton Hubble	a99a990d49	Remove unused Kubelet tls mounts * Kubelet trusts only the cluster CA certificate (and certificates in the Kubelet debian base image), there is no longer a need to mount the host's trusted certs * Similar change on Flatcar Linux in https://github.com/poseidon/typhoon/pull/855 Rel: https://github.com/poseidon/typhoon/pull/810	2020-10-18 23:48:21 -07:00
Dalton Hubble	df17253e72	Fix delete node permission on Fedora CoreOS node shutdown * On cloud platforms, `delete-node.service` tries to delete the local node (not always possible depending on preemption time) * Since v1.18.3, kubelet TLS bootstrap generates a kubeconfig in `/var/lib/kubelet` which should be used with kubectl in the delete-node oneshot	2020-10-18 23:38:11 -07:00
Dalton Hubble	eda78db08e	Change Flatcar kubelet.service container from rkt to docker * Use docker to run the `kubelet.service` container * Update Kubelet mounts to match Fedora CoreOS * Remove unused `/etc/ssl/certs` mount (see https://github.com/poseidon/typhoon/pull/810) * Remove unused `/usr/share/ca-certificates` mount * Remove `/etc/resolv.conf` mount, Docker default is ok * Change `delete-node.service` to use docker instead of rkt and inline ExecStart, as was done on Fedora CoreOS * Fix permission denied on shutdown `delete-node`, caused by the kubeconfig mount changing with the introduction of node TLS bootstrap Background * podmand, rkt, and runc daemonless container process runners provide advantages over the docker daemon for system containers. Docker requires workarounds for use in systemd units where the ExecStart must tail logs so systemd can monitor the daemonized container. https://github.com/moby/moby/issues/6791 * Why switch then? On Flatcar Linux, podman isn't shipped. rkt works, but isn't developing while container standards continue to move forward. Typhoon has used runc for the Kubelet runner before in Fedora Atomic, but its more low-level. So we're left with Docker, which is less than ideal, but shipped in Flatcar * Flatcar Linux appears to be shifting system components to use docker, which does provide some limited guards against breakages (e.g. Flatcar cannot enable docker live restore)	2020-10-18 23:24:45 -07:00
Dalton Hubble	afac46e39a	Remove asset_dir variable and optional asset writes * Originally, poseidon/terraform-render-bootstrap generated TLS certificates, manifests, and cluster "assets" written to local disk (`asset_dir`) during terraform apply cluster bootstrap * Typhoon v1.17.0 introduced bootstrapping using only Terraform state to store cluster assets, to avoid ever writing sensitive materials to disk and improve automated use-cases. `asset_dir` was changed to optional and defaulted to "" (no writes) * Typhoon v1.18.0 deprecated the `asset_dir` variable, removed docs, and announced it would be deleted in future. * Add Terraform output `assets_dir` map * Remove the `asset_dir` variable Cluster assets are now stored in Terraform state only. For those who wish to write those assets to local files, this is possible doing so explicitly. ``` resource local_file "assets" { for_each = module.yavin.assets_dist filename = "some-assets/${each.key}" content = each.value } ``` Related: * https://github.com/poseidon/typhoon/pull/595 * https://github.com/poseidon/typhoon/pull/678	2020-10-17 15:00:15 -07:00