Reduce Calcio MTU on Fedora CoreOS Azure

* Change the Calico VXLAN interface for MTU from 1450 to 1410 * VXLAN on Azure should support MTU 1450. However, there is history where performance measures have shown that 1410 is needed to have expected performance. Flatcar Linux has the same MTU 1410 override and note * FCOS 31.20200323.3.2 was known to perform fine with 1450, but now in 31.20200517.3.0 the right value seems to be 1410
Fix Kubelet starting before hostname set on FCOS AWS
2025-08-03 20:41:33 +02:00 · 2020-06-19 00:24:56 -07:00 · 2020-06-19 00:19:54 -07:00 · 2020-06-19 00:12:13 -07:00 · 2020-06-18 01:03:37 -07:00 · 2020-06-17 19:53:19 -07:00
124 changed files with 896 additions and 870 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@ -1,33 +0,0 @@
-<!-- Fill in either the 'Bug' or 'Feature Request' section -->
-
-## Bug
-
-### Environment
-
-* Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: fedora-coreos, flatcar-linux
-* Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
-* Terraform: `terraform version` (reporting latest is **not** helpful)
-* Plugins: Provider plugin versions (reporting latest is **not** helpful)
-
-### Problem
-
-Describe the problem.
-
-### Desired Behavior
-
-Describe the goal.
-
-### Steps to Reproduce
-
-Provide clear steps to reproduce the issue unless already covered.
-
-## Feature Request
-
-### Feature
-
-Describe the feature and what problem it solves.
-
-### Tradeoffs
-
-What are the pros and cons of this feature? How will it be exercised and maintained?
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@ -0,0 +1,39 @@
+---
+name: Bug report
+about: Report a bug to improve the project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+<!-- READ: Issues are used to receive focused bug reports from users and to track planned future enhancements by the authors. Topics like cluster operation, support, debugging help, advice, and Kubernetes concepts are out of scope and should not use issues-->
+
+**Description**
+
+A clear and concise description of what the bug is.
+
+**Steps to Reproduce**
+
+Provide clear steps to reproduce the bug.
+
+- [ ] Relevant error messages if appropriate (concise, not a dump of everything).
+- [ ] Explored using a vanilla cluster from the [tutorials](https://typhoon.psdn.io/#documentation). Ruled out [customizations](https://typhoon.psdn.io/advanced/customization/).
+
+**Expected behavior**
+
+A clear and concise description of what you expected to happen.
+
+**Environment**
+
+* Platform: aws, azure, bare-metal, google-cloud, digital-ocean
+* OS: fedora-coreos, flatcar-linux (include release version)
+* Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
+* Terraform: `terraform version` (reporting latest is **not** helpful)
+* Plugins: Provider plugin versions (reporting latest is **not** helpful)
+
+**Possible Solution**
+
+<!-- Most bug reports should have some inkling about solutions. Otherwise, your report may be less of a bug and more of a support request (see top).-->
+
+Link to a PR or description.
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@ -0,0 +1,5 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Security
+    url: https://typhoon.psdn.io/topics/security/
+    about: Report security vulnerabilities
--- a/.github/issue_template.md
+++ b/.github/issue_template.md
@ -0,0 +1,15 @@
+<!-- READ: Issues are used to receive focused bug reports from users and to track planned future enhancements by the authors. Topics like cluster operation, support, debugging help, advice, and Kubernetes concepts are out of scope and should not use issues-->
+
+## Enhancement
+
+### Overview
+
+One paragraph explanation of the enhancement.
+
+### Motivation
+
+Describe the motivation and what problem this solves.
+
+### Tradeoffs
+
+What are the pros and cons of this feature? How will it be exercised and maintained?
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,115 @@ Notable changes between versions.

 ## Latest

+## v1.18.4
+
+* Kubernetes [v1.18.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1184)
+* Update Kubelet image publishing ([#749](https://github.com/poseidon/typhoon/pull/749))
+  * Build Kubelet images internally and publish to Quay and Dockerhub
+    * [quay.io/poseidon/kubelet](https://quay.io/repository/poseidon/kubelet) (official)
+    * [docker.io/psdn/kubelet](https://hub.docker.com/r/psdn/kubelet) (fallback)
+  * Continue offering automated image builds with an alternate tag strategy (see [docs](https://typhoon.psdn.io/topics/security/#container-images))
+  * [Document](https://typhoon.psdn.io/advanced/customization/#kubelet) use of alternate Kubelet images during registry incidents
+* Update Calico from v3.14.0 to [v3.14.1](https://docs.projectcalico.org/v3.14/release-notes/)
+  * Fix [CVE-2020-13597](https://github.com/kubernetes/kubernetes/issues/91507)
+* Rename controller NoSchedule taint from `node-role.kubernetes.io/master` to `node-role.kubernetes.io/controller` ([#764](https://github.com/poseidon/typhoon/pull/764))
+  * Tolerate the new taint name for workloads that may run on controller nodes
+* Remove node label `node.kubernetes.io/master` from controller nodes ([#764](https://github.com/poseidon/typhoon/pull/764))
+  * Use `node.kubernetes.io/controller` (present since v1.9.5, [#160](https://github.com/poseidon/typhoon/pull/160)) to node select controllers
+* Remove unused Kubelet `-lock-file` and `-exit-on-lock-contention` ([#758](https://github.com/poseidon/typhoon/pull/758))
+
+### Fedora CoreOS
+
+#### Azure
+
+* Use `strict` Fedora CoreOS Config (FCC) snippet parsing ([#755](https://github.com/poseidon/typhoon/pull/755))
+* Reduce Calico vxlan interface MTU to maintain performance ([#767](https://github.com/poseidon/typhoon/pull/766))
+
+#### AWS
+
+* Fix Kubelet service race with hostname update ([#766](https://github.com/poseidon/typhoon/pull/766))
+  * Wait for a hostname to avoid Kubelet trying to register as `localhost`
+
+### Flatcar Linux
+
+* Use `strict` Container Linux Config (CLC) snippet parsing ([#755](https://github.com/poseidon/typhoon/pull/755))
+  * Require `terraform-provider-ct` v0.4+, recommend v0.5+ (**action required**)
+
+### Addons
+
+* Update nginx-ingress from v0.32.0 to [v0.33.0](https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.33.0)
+* Update Prometheus from v2.18.1 to [v2.19.0](https://github.com/prometheus/prometheus/releases/tag/v2.19.0)
+* Update node-exporter from v1.0.0-rc.1 to [v1.0.1](https://github.com/prometheus/node_exporter/releases/tag/v1.0.1)
+* Update kube-state-metrics from v1.9.6 to v1.9.7
+* Update Grafana from v7.0.0 to v7.0.3
+
+## v1.18.3
+
+* Kubernetes [v1.18.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183)
+* Use Kubelet [TLS bootstrap](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) with bootstrap token authentication ([#713](https://github.com/poseidon/typhoon/pull/713))
+  * Enable Node [Authorization](https://kubernetes.io/docs/reference/access-authn-authz/node/) and [NodeRestriction](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction) to reduce authorization scope
+  * Renew Kubelet certificates every 72 hours
+* Update etcd from v3.4.7 to [v3.4.9](https://github.com/etcd-io/etcd/releases/tag/v3.4.9)
+* Update Calico from v3.13.1 to [v3.14.0](https://docs.projectcalico.org/v3.14/release-notes/)
+* Add CoreDNS node affinity preference for controller nodes ([#188](https://github.com/poseidon/terraform-render-bootstrap/pull/188))
+* Deprecate CoreOS Container Linux support (no OS [updates](https://coreos.com/os/eol/) after May 2020)
+  * Use a `fedora-coreos` module for Fedora CoreOS
+  * Use a `container-linux` module for Flatcar Linux
+
+### AWS
+
+* Fix Terraform plan error when `controller_count` exceeds AWS zones (e.g. 5 controllers) ([#714](https://github.com/poseidon/typhoon/pull/714))
+  * Regressed in v1.17.1 ([#605](https://github.com/poseidon/typhoon/pull/605))
+
+### Azure
+
+* Update Azure subnets to set `address_prefixes` list ([#730](https://github.com/poseidon/typhoon/pull/730))
+  * Fix warning that `address_prefix` is deprecated
+  * Require `terraform-provider-azurerm` v2.8.0+ (action required)
+
+### DigitalOcean
+
+* Promote DigitalOcean to beta on both Fedora CoreOS and Flatcar Linux
+
+### Fedora CoreOS
+
+* Fix Calico `install-cni` crashloop on Pod restarts ([#724](https://github.com/poseidon/typhoon/pull/724))
+  * SELinux enforcement requires consistent file context MCS level
+  * Restarting a node resolved the issue as a previous workaround
+
+#### AWS
+
+* Support Fedora CoreOS [image streams](https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/) ([#727](https://github.com/poseidon/typhoon/pull/727))
+  * Add `os_stream` variable to set the stream to `stable` (default), `testing`, or `next`
+  * Remove unused `os_image` variable
+
+#### Google
+
+* Support Fedora CoreOS [image streams](https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/) ([#723](https://github.com/poseidon/typhoon/pull/722))
+  * Add `os_stream` variable to set the stream to `stable` (default), `testing`, or `next`
+  * Deprecate `os_image` variable. Manual image uploads are no longer needed
+
+### Flatcar Linux
+
+#### Azure
+
+* Use the Flatcar Linux Azure Marketplace image
+  * Restore [#664](https://github.com/poseidon/typhoon/pull/664) (reverted in [#707](https://github.com/poseidon/typhoon/pull/707)) but use Flatcar Linux new free offer (not byol)
+* Change `os_image` to use a `flatcar-stable` default
+
+#### Google
+
+* Promote Flatcar Linux to beta
+
+### Addons
+
+* Update nginx-ingress from v0.30.0 to [v0.32.0](https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.32.0)
+  * Add support for [IngressClass](https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class)
+* Update Prometheus from v2.17.1 to v2.18.1
+  * Update kube-state-metrics from v1.9.5 to [v1.9.6](https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.6)
+  * Update node-exporter from v1.0.0-rc.0 to [v1.0.0-rc.1](https://github.com/prometheus/node_exporter/releases/tag/v1.0.0-rc.1)
+* Update Grafana from v6.7.2 to [v7.0.0](https://grafana.com/docs/grafana/latest/guides/whats-new-in-v7-0/)
+
 ## v1.18.2

 * Kubernetes [v1.18.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1182)
--- a/README.md
+++ b/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)

@ -28,35 +28,25 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | stable |
 | Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](azure/fedora-coreos/kubernetes) | alpha |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
-| DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | alpha |
+| DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |

-Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org/releases/).
+Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
 | Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | alpha |
-| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | alpha |
-
-Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com/os/eol/) after May 2020).
-
-| Platform      | Operating System | Terraform Module | Status |
-|---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| Azure         | Container Linux  | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
+| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
+| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | beta |

 ## Documentation

 * [Docs](https://typhoon.psdn.io)
 * Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
 * Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Azure](docs/fedora-coreos/azure.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), [DigitalOcean](docs/fedora-coreos/digitalocean.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
-* Flatcar Linux tutorials for [AWS](docs/cl/aws.md), [Azure](docs/cl/azure.md), [Bare-Metal](docs/cl/bare-metal.md), [DigitalOcean](docs/cl/digital-ocean.md), and [Google Cloud](docs/cl/google-cloud.md)
+* Flatcar Linux tutorials for [AWS](docs/flatcar-linux/aws.md), [Azure](docs/flatcar-linux/azure.md), [Bare-Metal](docs/flatcar-linux/bare-metal.md), [DigitalOcean](docs/flatcar-linux/digitalocean.md), and [Google Cloud](docs/flatcar-linux/google-cloud.md)

 ## Usage

@ -64,7 +54,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.18.4"

  # Google Cloud
  cluster_name  = "yavin"
@ -103,9 +93,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.4
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.4
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.4
 ```

 List the pods.
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -23,7 +23,7 @@ spec:
    spec:
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:6.7.2
+          image: docker.io/grafana/grafana:7.0.3
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/class.yaml
+++ b/addons/nginx-ingress/aws/class.yaml
@ -0,0 +1,6 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: IngressClass
+metadata:
+  name: public
+spec:
+  controller: k8s.io/ingress-nginx
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/aws/rbac/cluster-role.yaml
+++ b/addons/nginx-ingress/aws/rbac/cluster-role.yaml
@ -51,3 +51,12 @@ rules:
      - ingresses/status
    verbs:
      - update
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+
--- a/addons/nginx-ingress/azure/class.yaml
+++ b/addons/nginx-ingress/azure/class.yaml
@ -0,0 +1,6 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: IngressClass
+metadata:
+  name: public
+spec:
+  controller: k8s.io/ingress-nginx
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/rbac/cluster-role.yaml
+++ b/addons/nginx-ingress/azure/rbac/cluster-role.yaml
@ -51,3 +51,12 @@ rules:
      - ingresses/status
    verbs:
      - update
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+
--- a/addons/nginx-ingress/bare-metal/class.yaml
+++ b/addons/nginx-ingress/bare-metal/class.yaml
@ -0,0 +1,6 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: IngressClass
+metadata:
+  name: public
+spec:
+  controller: k8s.io/ingress-nginx
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/rbac/cluster-role.yaml
+++ b/addons/nginx-ingress/bare-metal/rbac/cluster-role.yaml
@ -51,3 +51,12 @@ rules:
      - ingresses/status
    verbs:
      - update
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+
--- a/addons/nginx-ingress/digital-ocean/class.yaml
+++ b/addons/nginx-ingress/digital-ocean/class.yaml
@ -0,0 +1,6 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: IngressClass
+metadata:
+  name: public
+spec:
+  controller: k8s.io/ingress-nginx
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/rbac/cluster-role.yaml
+++ b/addons/nginx-ingress/digital-ocean/rbac/cluster-role.yaml
@ -51,3 +51,12 @@ rules:
      - ingresses/status
    verbs:
      - update
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+
--- a/addons/nginx-ingress/google-cloud/class.yaml
+++ b/addons/nginx-ingress/google-cloud/class.yaml
@ -0,0 +1,6 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: IngressClass
+metadata:
+  name: public
+spec:
+  controller: k8s.io/ingress-nginx
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/rbac/cluster-role.yaml
+++ b/addons/nginx-ingress/google-cloud/rbac/cluster-role.yaml
@ -51,3 +51,12 @@ rules:
      - ingresses/status
    verbs:
      - update
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -20,7 +20,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.17.1
+          image: quay.io/prometheus/prometheus:v2.19.0
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -24,7 +24,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.9.5
+        image: quay.io/coreos/kube-state-metrics:v1.9.7
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -28,7 +28,7 @@ spec:
      hostPID: true
      containers:
      - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.0.0-rc.0
+        image: quay.io/prometheus/node-exporter:v1.0.1
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -882,10 +882,10 @@ data:
            {
              "alert": "KubeClientCertificateExpiration",
              "annotations": {
-                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 7.0 days.",
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 1.0 hours.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
              },
-              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 604800\n",
+              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 3600\n",
              "labels": {
                "severity": "warning"
              }
@ -893,10 +893,10 @@ data:
            {
              "alert": "KubeClientCertificateExpiration",
              "annotations": {
-                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours.",
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 0.1 hours.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
              },
-              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 86400\n",
+              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 300\n",
              "labels": {
                "severity": "critical"
              }
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,11 +11,11 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Ready for Ingress, Prometheus, Grafana, CSI, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/container-linux/kubernetes/cl/controller.yaml
+++ b/aws/container-linux/kubernetes/cl/controller.yaml
@ -2,12 +2,12 @@
 systemd:
  units:
    - name: etcd-member.service
-      enable: true
+      enabled: true
      dropins:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.7"
+            Environment="ETCD_IMAGE_TAG=v3.4.9"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -28,11 +28,11 @@ systemd:
            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -46,12 +46,13 @@ systemd:
        RequiredBy=kubelet.service
        RequiredBy=etcd-member.service
    - name: kubelet.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -91,25 +92,24 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -134,7 +134,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -165,11 +165,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
@ -188,6 +188,7 @@ storage:
          done
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -36,7 +36,7 @@ resource "aws_instance" "controllers" {

  # network
  associate_public_ip_address = true
-  subnet_id                   = aws_subnet.public.*.id[count.index]
+  subnet_id                   = element(aws_subnet.public.*.id, count.index)
  vpc_security_group_ids      = [aws_security_group.controller.id]

  lifecycle {
@ -51,7 +51,7 @@ resource "aws_instance" "controllers" {
 data "ct_config" "controller-ignitions" {
  count    = var.controller_count
  content  = data.template_file.controller-configs.*.rendered[count.index]
-  pretty_print = false
+  strict   = true
  snippets = var.controller_snippets
 }

--- a/aws/container-linux/kubernetes/versions.tf
+++ b/aws/container-linux/kubernetes/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = "~> 0.12.6"
  required_providers {
    aws      = "~> 2.23"
-    ct       = "~> 0.3"
+    ct       = "~> 0.4"
    template = "~> 2.1"
    null     = "~> 2.1"
  }
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml
@ -2,11 +2,11 @@
 systemd:
  units:
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -19,12 +19,13 @@ systemd:
        [Install]
        RequiredBy=kubelet.service
    - name: kubelet.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -64,19 +65,18 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
@ -84,6 +84,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -112,6 +113,7 @@ storage:
          ${kubeconfig}
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
@ -127,7 +129,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -72,7 +72,7 @@ resource "aws_launch_configuration" "worker" {
 # Worker Ignition config
 data "ct_config" "worker-ignition" {
  content  = data.template_file.worker-config.rendered
-  pretty_print = false
+  strict   = true
  snippets = var.snippets
 }

--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,11 +11,11 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Ready for Ingress, Prometheus, Grafana, CSI, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -13,16 +13,8 @@ data "aws_ami" "fedora-coreos" {
    values = ["hvm"]
  }

-  filter {
-    name   = "name"
-    values = ["fedora-coreos-31.*.*.*-hvm"]
-  }
-
  filter {
    name   = "description"
-    values = ["Fedora CoreOS stable*"]
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
-
-  # try to filter out dev images (AWS filters can't)
-  name_regex = "^fedora-coreos-31.[0-9]*.[0-9]*.[0-9]*-hvm*"
 }
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/fedora-coreos/kubernetes/controllers.tf
+++ b/aws/fedora-coreos/kubernetes/controllers.tf
@ -36,7 +36,7 @@ resource "aws_instance" "controllers" {

  # network
  associate_public_ip_address = true
-  subnet_id                   = aws_subnet.public.*.id[count.index]
+  subnet_id                   = element(aws_subnet.public.*.id, count.index)
  vpc_security_group_ids      = [aws_security_group.controller.id]

  lifecycle {
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.7
+          quay.io/coreos/etcd:v3.4.9
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -38,11 +38,12 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Wait for DNS entries
+        Description=Wait for DNS and hostname
        Before=kubelet.service
        [Service]
        Type=oneshot
        RemainAfterExit=true
+        ExecStartPre=/bin/sh -c 'while [ `hostname -s` == "localhost" ]; do sleep 1; done;'
        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
        [Install]
        RequiredBy=kubelet.service
@ -51,9 +52,10 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -79,10 +81,11 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -90,16 +93,14 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -123,7 +124,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.2
+            quay.io/poseidon/kubelet:v1.18.4
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -151,11 +152,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -41,9 +41,9 @@ variable "worker_type" {
  default     = "t3.small"
 }

-variable "os_image" {
+variable "os_stream" {
  type        = string
-  description = "AMI channel for Fedora CoreOS (not yet used)"
+  description = "Fedora CoreOs image stream for instances (e.g. stable, testing, next)"
  default     = "stable"
 }

--- a/aws/fedora-coreos/kubernetes/workers.tf
+++ b/aws/fedora-coreos/kubernetes/workers.tf
@ -8,7 +8,7 @@ module "workers" {
  security_groups = [aws_security_group.worker.id]
  worker_count    = var.worker_count
  instance_type   = var.worker_type
-  os_image        = var.os_image
+  os_stream       = var.os_stream
  disk_size       = var.disk_size
  spot_price      = var.worker_price
  target_groups   = var.worker_target_groups
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -13,16 +13,8 @@ data "aws_ami" "fedora-coreos" {
    values = ["hvm"]
  }

-  filter {
-    name   = "name"
-    values = ["fedora-coreos-31.*.*.*-hvm"]
-  }
-
  filter {
    name   = "description"
-    values = ["Fedora CoreOS stable*"]
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
-
-  # try to filter out dev images (AWS filters can't)
-  name_regex = "^fedora-coreos-31.[0-9]*.[0-9]*.[0-9]*-hvm*"
 }
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -9,11 +9,12 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Wait for DNS entries
+        Description=Wait for DNS and hostname
        Before=kubelet.service
        [Service]
        Type=oneshot
        RemainAfterExit=true
+        ExecStartPre=/bin/sh -c 'while [ `hostname -s` == "localhost" ]; do sleep 1; done;'
        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
        [Install]
        RequiredBy=kubelet.service
@ -21,9 +22,10 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -49,10 +51,11 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -60,10 +63,8 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
@ -71,6 +72,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -87,7 +89,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.4 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/aws/fedora-coreos/kubernetes/workers/variables.tf
+++ b/aws/fedora-coreos/kubernetes/workers/variables.tf
@ -34,9 +34,9 @@ variable "instance_type" {
  default     = "t3.small"
 }

-variable "os_image" {
+variable "os_stream" {
  type        = string
-  description = "AMI channel for Fedora CoreOS (not yet used)"
+  description = "Fedora CoreOs image stream for instances (e.g. stable, testing, next)"
  default     = "stable"
 }

--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/azure/container-linux/kubernetes/bootstrap.tf
+++ b/azure/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/container-linux/kubernetes/cl/controller.yaml
+++ b/azure/container-linux/kubernetes/cl/controller.yaml
@ -2,12 +2,12 @@
 systemd:
  units:
    - name: etcd-member.service
-      enable: true
+      enabled: true
      dropins:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.7"
+            Environment="ETCD_IMAGE_TAG=v3.4.9"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -28,11 +28,11 @@ systemd:
            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -46,12 +46,13 @@ systemd:
        RequiredBy=kubelet.service
        RequiredBy=etcd-member.service
    - name: kubelet.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -90,24 +91,23 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -163,11 +163,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
@ -186,6 +186,7 @@ storage:
          done
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -53,18 +53,24 @@ resource "azurerm_linux_virtual_machine" "controllers" {
    storage_account_type = "Premium_LRS"
  }

-  // CoreOS Container Linux or Flatcar Container Linux (manual upload)
-  dynamic "source_image_reference" {
-    for_each = local.flavor == "coreos" ? [1] : []
-
-    content {
-      publisher = "CoreOS"
-      offer     = "CoreOS"
+  # CoreOS Container Linux or Flatcar Container Linux
+  source_image_reference {
+    publisher = local.flavor == "flatcar" ? "Kinvolk" : "CoreOS"
+    offer     = local.flavor == "flatcar" ? "flatcar-container-linux-free" : "CoreOS"
    sku       = local.channel
    version   = "latest"
  }
+
+  # Gross hack for Flatcar Linux
+  dynamic "plan" {
+    for_each = local.flavor == "flatcar" ? [1] : []
+
+    content {
+      name      = local.channel
+      publisher = "kinvolk"
+      product   = "flatcar-container-linux-free"
+    }
  }
-  source_image_id = local.flavor == "coreos" ? null : var.os_image

  # network
  network_interface_ids = [
@ -135,7 +141,7 @@ resource "azurerm_network_interface_backend_address_pool_association" "controlle
 data "ct_config" "controller-ignitions" {
  count    = var.controller_count
  content  = data.template_file.controller-configs.*.rendered[count.index]
-  pretty_print = false
+  strict   = true
  snippets = var.controller_snippets
 }

--- a/azure/container-linux/kubernetes/network.tf
+++ b/azure/container-linux/kubernetes/network.tf
@ -21,7 +21,7 @@ resource "azurerm_subnet" "controller" {

  name                 = "controller"
  virtual_network_name = azurerm_virtual_network.network.name
-  address_prefix       = cidrsubnet(var.host_cidr, 1, 0)
+  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 0)]
 }

 resource "azurerm_subnet_network_security_group_association" "controller" {
@ -34,7 +34,7 @@ resource "azurerm_subnet" "worker" {

  name                 = "worker"
  virtual_network_name = azurerm_virtual_network.network.name
-  address_prefix       = cidrsubnet(var.host_cidr, 1, 1)
+  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 1)]
 }

 resource "azurerm_subnet_network_security_group_association" "worker" {
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -48,7 +48,8 @@ variable "worker_type" {

 variable "os_image" {
  type        = string
-  description = "Channel for a Container Linux derivative (/subscriptions/some-flatcar-upload, coreos-stable, coreos-beta, coreos-alpha)"
+  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge, coreos-stable, coreos-beta, coreos-alpha)"
+  default     = "flatcar-stable"
 }

 variable "disk_size" {
--- a/azure/container-linux/kubernetes/versions.tf
+++ b/azure/container-linux/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = "~> 0.12.6"
  required_providers {
-    azurerm  = "~> 2.0"
-    ct       = "~> 0.3"
+    azurerm  = "~> 2.8"
+    ct       = "~> 0.4"
    template = "~> 2.1"
    null     = "~> 2.1"
  }
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml
@ -2,11 +2,11 @@
 systemd:
  units:
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -19,12 +19,13 @@ systemd:
        [Install]
        RequiredBy=kubelet.service
    - name: kubelet.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -63,18 +64,17 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
@ -82,6 +82,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -89,7 +90,7 @@ systemd:
        [Install]
        WantedBy=multi-user.target
    - name: delete-node.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Waiting to delete Kubernetes node on shutdown
@ -110,6 +111,7 @@ storage:
          ${kubeconfig}
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
@ -125,7 +127,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@ -46,7 +46,7 @@ variable "vm_type" {

 variable "os_image" {
  type        = string
-  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, coreos-stable, coreos-beta, coreos-alpha)"
+  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge, coreos-stable, coreos-beta, coreos-alpha)"
  default     = "flatcar-stable"
 }

--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -24,18 +24,24 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
    caching              = "ReadWrite"
  }

-  // CoreOS Container Linux or Flatcar Container Linux (manual upload)
-  dynamic "source_image_reference" {
-    for_each = local.flavor == "coreos" ? [1] : []
-
-    content {
-      publisher = "CoreOS"
-      offer     = "CoreOS"
+  # CoreOS Container Linux or Flatcar Container Linux
+  source_image_reference {
+    publisher = local.flavor == "flatcar" ? "Kinvolk" : "CoreOS"
+    offer     = local.flavor == "flatcar" ? "flatcar-container-linux-free" : "CoreOS"
    sku       = local.channel
    version   = "latest"
  }
+
+  # Gross hack for Flatcar Linux
+  dynamic "plan" {
+    for_each = local.flavor == "flatcar" ? [1] : []
+
+    content {
+      name      = local.channel
+      publisher = "kinvolk"
+      product   = "flatcar-container-linux-free"
+    }
  }
-  source_image_id = local.flavor == "coreos" ? null : var.os_image

  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
  admin_username = "core"
@ -92,7 +98,7 @@ resource "azurerm_monitor_autoscale_setting" "workers" {
 # Worker Ignition configs
 data "ct_config" "worker-ignition" {
  content  = data.template_file.worker-config.rendered
-  pretty_print = false
+  strict   = true
  snippets = var.snippets
 }

--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -10,8 +10,9 @@ module "bootstrap" {
  networking = var.networking

  # only effective with Calico networking
+  # we should be able to use 1450 MTU, but in practice, 1410 was needed
  network_encapsulation = "vxlan"
-  network_mtu           = "1450"
+  network_mtu           = "1410"

  pod_cidr              = var.pod_cidr
  service_cidr          = var.service_cidr
--- a/azure/fedora-coreos/kubernetes/controllers.tf
+++ b/azure/fedora-coreos/kubernetes/controllers.tf
@ -115,7 +115,7 @@ resource "azurerm_network_interface_backend_address_pool_association" "controlle
 data "ct_config" "controller-ignitions" {
  count    = var.controller_count
  content  = data.template_file.controller-configs.*.rendered[count.index]
-  pretty_print = false
+  strict   = true
  snippets = var.controller_snippets
 }

--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.7
+          quay.io/coreos/etcd:v3.4.9
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -51,9 +51,10 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -79,10 +80,11 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -90,16 +92,14 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -123,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.2
+            quay.io/poseidon/kubelet:v1.18.4
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -151,11 +151,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
--- a/azure/fedora-coreos/kubernetes/network.tf
+++ b/azure/fedora-coreos/kubernetes/network.tf
@ -21,7 +21,7 @@ resource "azurerm_subnet" "controller" {

  name                 = "controller"
  virtual_network_name = azurerm_virtual_network.network.name
-  address_prefix       = cidrsubnet(var.host_cidr, 1, 0)
+  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 0)]
 }

 resource "azurerm_subnet_network_security_group_association" "controller" {
@ -34,7 +34,7 @@ resource "azurerm_subnet" "worker" {

  name                 = "worker"
  virtual_network_name = azurerm_virtual_network.network.name
-  address_prefix       = cidrsubnet(var.host_cidr, 1, 1)
+  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 1)]
 }

 resource "azurerm_subnet_network_security_group_association" "worker" {
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = "~> 0.12.6"
  required_providers {
-    azurerm  = "~> 2.0"
-    ct       = "~> 0.3"
+    azurerm  = "~> 2.8"
+    ct       = "~> 0.4"
    template = "~> 2.1"
    null     = "~> 2.1"
  }
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -21,9 +21,10 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -49,10 +50,11 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -60,10 +62,8 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
@ -71,6 +71,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -87,7 +88,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.4 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/azure/fedora-coreos/kubernetes/workers/workers.tf
+++ b/azure/fedora-coreos/kubernetes/workers/workers.tf
@ -73,7 +73,7 @@ resource "azurerm_monitor_autoscale_setting" "workers" {
 # Worker Ignition configs
 data "ct_config" "worker-ignition" {
  content  = data.template_file.worker-config.rendered
-  pretty_print = false
+  strict   = true
  snippets = var.snippets
 }

--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/container-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml
@ -2,12 +2,12 @@
 systemd:
  units:
    - name: etcd-member.service
-      enable: true
+      enabled: true
      dropins:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.7"
+            Environment="ETCD_IMAGE_TAG=v3.4.9"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -28,11 +28,11 @@ systemd:
            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: kubelet.path
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Watch for kubeconfig
@ -41,7 +41,7 @@ systemd:
        [Install]
        WantedBy=multi-user.target
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -57,9 +57,10 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -103,26 +104,25 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -147,7 +147,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -158,6 +158,7 @@ storage:
  directories:
    - path: /etc/kubernetes
      filesystem: root
+      mode: 0755
  files:
    - path: /etc/hostname
      filesystem: root
@ -181,11 +182,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
@ -204,6 +205,7 @@ storage:
          done
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
--- a/bare-metal/container-linux/kubernetes/cl/install.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/install.yaml
@ -2,7 +2,7 @@
 systemd:
  units:
    - name: installer.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Requires=network-online.target
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml
@ -2,11 +2,11 @@
 systemd:
  units:
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: kubelet.path
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Watch for kubeconfig
@ -15,7 +15,7 @@ systemd:
        [Install]
        WantedBy=multi-user.target
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -30,9 +30,10 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -76,20 +77,19 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in compact(split(",", node_labels)) ~}
@ -100,6 +100,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -111,6 +112,7 @@ storage:
  directories:
    - path: /etc/kubernetes
      filesystem: root
+      mode: 0755
  files:
    - path: /etc/hostname
      filesystem: root
@ -120,6 +122,7 @@ storage:
          ${domain_name}
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -143,7 +143,7 @@ resource "matchbox_profile" "controllers" {
 data "ct_config" "controller-ignitions" {
  count    = length(var.controllers)
  content  = data.template_file.controller-configs.*.rendered[count.index]
-  pretty_print = false
+  strict   = true
  snippets = lookup(var.snippets, var.controllers.*.name[count.index], [])
 }

@ -173,7 +173,7 @@ resource "matchbox_profile" "workers" {
 data "ct_config" "worker-ignitions" {
  count    = length(var.workers)
  content  = data.template_file.worker-configs.*.rendered[count.index]
-  pretty_print = false
+  strict   = true
  snippets = lookup(var.snippets, var.workers.*.name[count.index], [])
 }

--- a/bare-metal/container-linux/kubernetes/versions.tf
+++ b/bare-metal/container-linux/kubernetes/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = "~> 0.12.6"
  required_providers {
    matchbox = "~> 0.3.0"
-    ct       = "~> 0.3"
+    ct       = "~> 0.4"
    template = "~> 2.1"
    null     = "~> 2.1"
  }
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.7
+          quay.io/coreos/etcd:v3.4.9
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -50,9 +50,10 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -80,10 +81,11 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -91,17 +93,15 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -134,7 +134,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.2
+            quay.io/poseidon/kubelet:v1.18.4
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -162,11 +162,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -20,9 +20,10 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -50,10 +51,11 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -61,11 +63,9 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in compact(split(",", node_labels)) ~}
@ -76,6 +76,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/digital-ocean/container-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml
@ -2,12 +2,12 @@
 systemd:
  units:
    - name: etcd-member.service
-      enable: true
+      enabled: true
      dropins:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.7"
+            Environment="ETCD_IMAGE_TAG=v3.4.9"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -28,11 +28,11 @@ systemd:
            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: kubelet.path
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Watch for kubeconfig
@ -41,7 +41,7 @@ systemd:
        [Install]
        WantedBy=multi-user.target
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -57,11 +57,12 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Requires=coreos-metadata.service
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -101,25 +102,24 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -144,7 +144,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -155,6 +155,7 @@ storage:
  directories:
    - path: /etc/kubernetes
      filesystem: root
+      mode: 0755
  files:
    - path: /opt/bootstrap/layout
      filesystem: root
@ -172,11 +173,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
@ -195,6 +196,7 @@ storage:
          done
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml
@ -2,11 +2,11 @@
 systemd:
  units:
    - name: docker.service
-      enable: true
+      enabled: true
    - name: locksmithd.service
      mask: true
    - name: kubelet.path
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Watch for kubeconfig
@ -15,7 +15,7 @@ systemd:
        [Install]
        WantedBy=multi-user.target
    - name: wait-for-dns.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Wait for DNS entries
@ -30,11 +30,12 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube
+        Description=Kubelet
        Requires=coreos-metadata.service
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.4
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -74,23 +75,23 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.2 -- \
+          $${KUBELET_IMAGE} -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -98,7 +99,7 @@ systemd:
        [Install]
        WantedBy=multi-user.target
    - name: delete-node.service
-      enable: true
+      enabled: true
      contents: |
        [Unit]
        Description=Waiting to delete Kubernetes node on shutdown
@ -113,9 +114,11 @@ storage:
  directories:
    - path: /etc/kubernetes
      filesystem: root
+      mode: 0755
  files:
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
+      mode: 0644
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
@ -131,7 +134,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.2 \
+            docker://quay.io/poseidon/kubelet:v1.18.4 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/controllers.tf
+++ b/digital-ocean/container-linux/kubernetes/controllers.tf
@ -71,7 +71,7 @@ resource "digitalocean_tag" "controllers" {
 data "ct_config" "controller-ignitions" {
  count    = var.controller_count
  content  = data.template_file.controller-configs.*.rendered[count.index]
-  pretty_print = false
+  strict   = true
  snippets = var.controller_snippets
 }

--- a/digital-ocean/container-linux/kubernetes/versions.tf
+++ b/digital-ocean/container-linux/kubernetes/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = "~> 0.12.6"
  required_providers {
    digitalocean = "~> 1.3"
-    ct           = "~> 0.3"
+    ct           = "~> 0.4"
    template     = "~> 2.1"
    null         = "~> 2.1"
  }
--- a/digital-ocean/container-linux/kubernetes/workers.tf
+++ b/digital-ocean/container-linux/kubernetes/workers.tf
@ -59,7 +59,7 @@ resource "digitalocean_tag" "workers" {
 # Worker Ignition config
 data "ct_config" "worker-ignition" {
  content  = data.template_file.worker-config.rendered
-  pretty_print = false
+  strict   = true
  snippets = var.worker_snippets
 }

--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, and other [addons](https://typhoon.psdn.io/addons/overview/)

--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=14d0b2087962a0f2557c184f3f523548ce19bbdc"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e75697ce35d7773705f0b9b28ce1ffbe99f9493c"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.7
+          quay.io/coreos/etcd:v3.4.9
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -50,11 +50,12 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Requires=afterburn.service
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -81,10 +82,11 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -92,17 +94,15 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -135,7 +135,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.2
+            quay.io/poseidon/kubelet:v1.18.4
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -158,11 +158,11 @@ storage:
          chmod -R 500 /etc/ssl/etcd
          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          sudo mkdir -p /etc/kubernetes/manifests
-          sudo mv static-manifests/* /etc/kubernetes/manifests/
-          sudo mkdir -p /opt/bootstrap/assets
-          sudo mv manifests /opt/bootstrap/assets/manifests
-          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -21,11 +21,12 @@ systemd:
      enabled: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube (System Container)
+        Description=Kubelet (System Container)
        Requires=afterburn.service
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.4
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -52,10 +53,11 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.2 \
+          $${KUBELET_IMAGE} \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
          --enforce-node-allocatable=pods \
@ -63,15 +65,14 @@ systemd:
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --exit-on-lock-contention \
          --healthz-port=0 \
          --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
-          --kubeconfig=/etc/kubernetes/kubeconfig \
-          --lock-file=/var/run/lock/kubelet.lock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
@ -97,7 +98,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.2 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.4 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/digital-ocean/fedora-coreos/kubernetes/versions.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = "~> 0.12.6"
  required_providers {
    digitalocean = "~> 1.3"
-    ct           = "~> 0.3"
+    ct           = "~> 0.4"
    template     = "~> 2.1"
    null         = "~> 2.1"
  }
--- a/docs/advanced/customization.md
+++ b/docs/advanced/customization.md
@ -174,3 +174,34 @@ module "nemo" {

 To customize low-level Kubernetes control plane bootstrapping, see the [poseidon/terraform-render-bootstrap](https://github.com/poseidon/terraform-render-bootstrap) Terraform module.

+## Kubelet
+
+Typhoon publishes Kubelet [container images](/topics/security/#container-images) to Quay.io (default) and to Dockerhub (in case of a Quay [outage](https://github.com/poseidon/typhoon/issues/735) or breach). Quay automated builds also provide the option for fully verifiable tagged images (`build-{short_sha}`).
+
+To set an alternative Kubelet image, use a snippet to set a systemd dropin.
+
+```
+# host-image-override.yaml
+variant: fcos           <- remove for Flatcar Linux
+version: 1.0.0          <- remove for Flatcar Linux
+systemd:
+  units:
+    - name: kubelet.service
+      dropins:
+        - name: 10-image-override.conf
+          contents: |
+            [Service]
+            Environment=KUBELET_IMAGE=docker.io/psdn/kubelet:v1.18.3
+```
+
+```
+module "nemo" {
+  ...
+
+  worker_snippets = [
+    file("./snippets/host-image-override.yaml")
+  ]
+  ...
+}
+```
+
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -13,7 +13,7 @@ Internal Terraform Modules:

 ## AWS

-Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a worker pool using the AWS internal `workers` module.
+Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster). Define a worker pool using the AWS internal `workers` module.

 ```tf
 module "tempest-worker-pool" {
@ -78,11 +78,11 @@ Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-typ

 ## Azure

-Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define a worker pool using the Azure internal `workers` module.
+Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluster). Define a worker pool using the Azure internal `workers` module.

 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.18.4"

  # Azure
  region                  = module.ramius.region
@ -134,7 +134,7 @@ The Azure internal `workers` module supports a number of [variables](https://git
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | vm_type | Machine type for instances | "Standard_DS1_v2" | See below |
-| os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, coreos-stable, coreos-beta, coreos-alpha |
+| os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge, coreos-stable, coreos-beta, coreos-alpha |
 | priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | "Regular" | "Spot" |
 | snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
@ -144,11 +144,11 @@ Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricin

 ## Google Cloud

-Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#cluster). Define a worker pool using the Google Cloud internal `workers` module.
+Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-cloud.md#cluster). Define a worker pool using the Google Cloud internal `workers` module.

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.18.4"

  # Google Cloud
  region       = "europe-west2"
@ -179,11 +179,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.18.2
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.18.2
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.18.2
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.18.2
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.18.2
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.18.4
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.18.4
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.18.4
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.18.4
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.18.4
 ```

 ### Variables
@ -210,6 +210,7 @@ Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-z
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | machine_type | Compute instance machine type | "n1-standard-1" | See below |
+| os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | preemptible | If true, Compute Engine will terminate instances randomly within 24 hours | false | true |
 | snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
--- a/docs/architecture/operating-systems.md
+++ b/docs/architecture/operating-systems.md
@ -1,6 +1,6 @@
 # Operating Systems

-Typhoon supports [Fedora CoreOS](https://getfedora.org/coreos/), [Flatcar Linux](https://www.flatcar-linux.org/) and Container Linux (EOL in May 2020). These operating systems were chosen because they offer:
+Typhoon supports [Fedora CoreOS](https://getfedora.org/coreos/) and [Flatcar Linux](https://www.flatcar-linux.org/). These operating systems were chosen because they offer:

 * Minimalism and focus on clustered operation
 * Automated and atomic operating system upgrades
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -49,7 +49,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf

 ```tf
 provider "aws" {
-  version                 = "2.53.0"
+  version                 = "2.66.0"
  region                  = "eu-central-1"
  shared_credentials_file = "/home/user/.config/aws/credentials"
 }
@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.18.4"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.18.2
-ip-10-0-26-65  Ready   <none>   10m  v1.18.2
-ip-10-0-41-21  Ready   <none>   10m  v1.18.2
+ip-10-0-3-155  Ready   <none>   10m  v1.18.4
+ip-10-0-26-65  Ready   <none>   10m  v1.18.4
+ip-10-0-41-21  Ready   <none>   10m  v1.18.4
 ```

 List the pods.
@ -208,7 +208,7 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
-| os_image | AMI channel for Fedora CoreOS | not yet used | ? |
+| os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
 | disk_size | Size of the EBS volume in GB | 40 | 100 |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
--- a/docs/fedora-coreos/azure.md
+++ b/docs/fedora-coreos/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on Azure with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -47,7 +47,7 @@ Configure the Azure provider in a `providers.tf` file.

 ```tf
 provider "azurerm" {
-  version = "2.5.0"
+  version = "2.14.0"
 }

 provider "ct" {
@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.18.4"

  # Azure
  cluster_name   = "ramius"
@ -158,9 +158,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.18.2
-ramius-worker-000001  Ready   <none>  25m  v1.18.2
-ramius-worker-000002  Ready   <none>  24m  v1.18.2
+ramius-controller-0   Ready   <none>  24m  v1.18.4
+ramius-worker-000001  Ready   <none>  25m  v1.18.4
+ramius-worker-000002  Ready   <none>  24m  v1.18.4
 ```

 List the pods.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.18.2 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.18.4 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.4"

  # bare-metal
  cluster_name            = "mercury"
@ -289,9 +289,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.18.2
-node2.example.com   Ready   <none>  10m  v1.18.2
-node3.example.com   Ready   <none>  10m  v1.18.2
+node1.example.com   Ready   <none>  10m  v1.18.4
+node2.example.com   Ready   <none>  10m  v1.18.4
+node3.example.com   Ready   <none>  10m  v1.18.4
 ```

 List the pods.
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@ -1,6 +1,6 @@
-# Digital Ocean
+# DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on DigitalOcean with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -50,7 +50,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.

 ```tf
 provider "digitalocean" {
-  version = "1.15.1"
+  version = "1.20.0"
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
 }

@ -79,7 +79,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.18.4"

  # Digital Ocean
  cluster_name = "nemo"
@ -153,9 +153,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.18.2
-10.132.115.81      Ready   <none>  10m  v1.18.2
-10.132.124.107     Ready   <none>  10m  v1.18.2
+10.132.110.130     Ready   <none>  10m  v1.18.4
+10.132.115.81      Ready   <none>  10m  v1.18.4
+10.132.124.107     Ready   <none>  10m  v1.18.4
 ```

 List the pods.
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on Google Compute Engine with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -49,7 +49,7 @@ Configure the Google Cloud provider to use your service account key, project-id,

 ```tf
 provider "google" {
-  version     = "3.12.0"
+  version     = "3.26.0"
  project     = "project-id"
  region      = "us-central1"
  credentials = file("~/.config/google-cloud/terraform.json")
@ -65,25 +65,6 @@ Additional configuration options are described in the `google` provider [docs](h
 !!! tip
    Regions are listed in [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`. A project may contain multiple clusters across different regions.

-## Fedora CoreOS Images
-
-Fedora CoreOS publishes images for Google Cloud, but does not yet upload them. Google Cloud allows [custom boot images](https://cloud.google.com/compute/docs/images/import-existing-image) to be uploaded to a bucket and imported into your project.
-
-[Download](https://getfedora.org/coreos/download/) a Fedora CoreOS GCP gzipped tarball and upload it to a Google Cloud storage bucket.
-
-```
-gsutil list
-gsutil cp fedora-coreos-31.20200323.3.2-gcp.x86_64.tar.gz gs://BUCKET
-```
-
-Create a Compute Engine image from the file.
-
-```
-gcloud compute images create fedora-coreos-31-20200323-3-2 --source-uri gs://BUCKET/fedora-coreos-31.20200323.3.2-gcp.x86_64.tar.gz
-```
-
-Set the [os_image](#variables) in the next step.
-
 ## Cluster

 Define a Kubernetes cluster using the module `google-cloud/fedora-coreos/kubernetes`.
@ -99,7 +80,6 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # configuration
-  os_image           = "fedora-coreos-31-20200323-3-2"
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."

  # optional
@ -165,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.4
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.4
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.4
 ```

 List the pods.
@ -204,7 +184,6 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-
 | region | Google Cloud region | "us-central1" |
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
-| os_image | Fedora CoreOS image for compute instances | "fedora-coreos-31-20200323-3-2" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

 Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Fedora CoreOS [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep fedora-coreos`.
@ -234,6 +213,7 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
+| os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
--- a/docs/flatcar-linux/aws.md
+++ b/docs/flatcar-linux/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on AWS with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on AWS with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -49,7 +49,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf

 ```tf
 provider "aws" {
-  version                 = "2.53.0"
+  version                 = "2.66.0"
  region                  = "eu-central-1"
  shared_credentials_file = "/home/user/.config/aws/credentials"
 }
@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.18.4"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.18.2
-ip-10-0-26-65  Ready   <none>  10m  v1.18.2
-ip-10-0-41-21  Ready   <none>  10m  v1.18.2
+ip-10-0-3-155  Ready   <none>  10m  v1.18.4
+ip-10-0-26-65  Ready   <none>  10m  v1.18.4
+ip-10-0-41-21  Ready   <none>  10m  v1.18.4
 ```

 List the pods.
--- a/docs/flatcar-linux/azure.md
+++ b/docs/flatcar-linux/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Azure with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on Azure with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -47,7 +47,7 @@ Configure the Azure provider in a `providers.tf` file.

 ```tf
 provider "azurerm" {
-  version = "2.5.0"
+  version = "2.14.0"
 }

 provider "ct" {
@ -57,45 +57,22 @@ provider "ct" {

 Additional configuration options are described in the `azurerm` provider [docs](https://www.terraform.io/docs/providers/azurerm/).

-### Flatcar Linux Images
+## Flatcar Linux Images

-Flatcar Linux publishes images for Azure. Azure allows custom images to be uploaded to a storage account bucket and imported.
-
-[Download](https://www.flatcar-linux.org/releases/) a Flatcar Linux Azure VHD image and upload it to an Azure storage account container (i.e. bucket).
-
-Azure requires fixed VHDs and Flatcar Linux provides dynamic VHDs, so uploads require Azure tools and cannot be done through the UI. Azure's tool compilation requires old versions, so Flatcar Linux has packaged a container image you may choose to use. See their [docs](https://docs.flatcar-linux.org/os/booting-on-azure/#uploading-your-own-image).
+Flatcar Linux publishes images to the Azure Marketplace and requires accepting terms.

 ```
-bzip2 -d flatcar_production_azure_image.vhd.bz2
+az vm image terms show --publish kinvolk --offer flatcar-container-linux-free --plan stable
+az vm image terms accept --publish kinvolk --offer flatcar-container-linux-free --plan stable
 ```

-```
-podman run -it --entrypoint=/bin/bash quay.io/kinvolk/azure-flatcar-image-upload
-...
-
-# az login
-# az storage account keys list --resource-group GROUP	--account-name BUCKET | jq -r '.[0].value'
-# azure-vhd-utils upload --localvhdpath /data/flatcar_production_azure_image.vhd --stgaccountname BUCKET --containername flatcar-linux --blobname flatcar-stable-2345.3.1 --stgaccountkey "KEYFROMABOVE"
-# exit
-```
-
-Create an Azure disk (note disk ID) and create an Azure image from it (note image ID).
-
-```
-az disk create --name flatcar-stable-2345.3.1 -g GROUP --source https://BUCKET.blob.core.windows.net/flatcar-linux/flatcar_production_azure_image.vhd
-
-az image create --name flatcar-stable-2345.3.1 -g GROUP --os-type=linux --source /subscriptions/some/path/providers/Microsoft.Compute/disks/flatcar-stable-2345.3.1
-```
-
-Set the [os_image](#variables) in the next step.
-
 ## Cluster

 Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.18.4"

  # Azure
  cluster_name   = "ramius"
@ -104,7 +81,6 @@ module "ramius" {
  dns_zone_group = "example-group"

  # configuration
-  os_image           = "/subscriptions/some/path/Microsoft.Compute/images/flatcar-stable-2345.3.1"
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."

  # optional
@ -115,15 +91,6 @@ module "ramius" {

 Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/container-linux/kubernetes/variables.tf) source.

-### Flatcar Linux Only
-
-Flatcar Linux publishes images to the Azure Marketplace and requires accepting their legal terms.
-
-```
-az vm image terms show --publish kinvolk --offer flatcar-container-linux --plan stable
-az vm image terms accept --publish kinvolk --offer flatcar-container-linux --plan stable
-```
-
 ## ssh-agent

 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
@ -179,9 +146,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.18.2
-ramius-worker-000001  Ready   <none>  25m  v1.18.2
-ramius-worker-000002  Ready   <none>  24m  v1.18.2
+ramius-controller-0   Ready   <none>  24m  v1.18.4
+ramius-worker-000001  Ready   <none>  25m  v1.18.4
+ramius-worker-000002  Ready   <none>  24m  v1.18.4
 ```

 List the pods.
@ -218,7 +185,6 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/c
 | region | Azure region | "centralus" |
 | dns_zone | Azure DNS zone | "azure.example.com" |
 | dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
-| os_image | Container Linux image for instances | "/subscriptions/..../some-flatcar-image", coreos-stable, coreos-beta, coreos-alpha |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

 !!! tip
@ -259,6 +225,7 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "Standard_B2s" | See below |
 | worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
+| os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge, coreos-stable, coreos-beta, coreos-alpha |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
--- a/docs/flatcar-linux/bare-metal.md
+++ b/docs/flatcar-linux/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.18.2 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.18.4 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.4"

  # bare-metal
  cluster_name            = "mercury"
@ -299,9 +299,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.18.2
-node2.example.com   Ready   <none>  10m  v1.18.2
-node3.example.com   Ready   <none>  10m  v1.18.2
+node1.example.com   Ready   <none>  10m  v1.18.4
+node2.example.com   Ready   <none>  10m  v1.18.4
+node3.example.com   Ready   <none>  10m  v1.18.4
 ```

 List the pods.
--- a/docs/flatcar-linux/digitalocean.md
+++ b/docs/flatcar-linux/digitalocean.md
@ -1,6 +1,6 @@
-# Digital Ocean
+# DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -50,7 +50,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.

 ```tf
 provider "digitalocean" {
-  version = "1.15.1"
+  version = "1.20.0"
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
 }

@ -79,7 +79,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.18.4"

  # Digital Ocean
  cluster_name = "nemo"
@ -153,9 +153,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.18.2
-10.132.115.81      Ready   <none>  10m  v1.18.2
-10.132.124.107     Ready   <none>  10m  v1.18.2
+10.132.110.130     Ready   <none>  10m  v1.18.4
+10.132.115.81      Ready   <none>  10m  v1.18.4
+10.132.124.107     Ready   <none>  10m  v1.18.4
 ```

 List the pods.
--- a/docs/flatcar-linux/google-cloud.md
+++ b/docs/flatcar-linux/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.18.2 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.4 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -49,7 +49,7 @@ Configure the Google Cloud provider to use your service account key, project-id,

 ```tf
 provider "google" {
-  version     = "3.12.0"
+  version     = "3.26.0"
  project     = "project-id"
  region      = "us-central1"
  credentials = file("~/.config/google-cloud/terraform.json")
@ -90,7 +90,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.4"

  # Google Cloud
  cluster_name  = "yavin"
@ -165,9 +165,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.4
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.4
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.4
 ```

 List the pods.
--- a/docs/index.md
+++ b/docs/index.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.2 (upstream)
+* Kubernetes v1.18.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
+* Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](addons/overview/)

 ## Modules
@ -28,35 +28,24 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | stable |
 | Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](fedora-coreos/azure.md) | alpha |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | beta |
-| DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](fedora-coreos/digitalocean.md) | alpha |
+| DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](fedora-coreos/digitalocean.md) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |

-Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org/releases/).
+Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](cl/aws.md) | stable |
-| Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
-| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | alpha |
-| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | alpha |
-
-Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com/os/eol/) after May 2020).
-
-| Platform      | Operating System | Terraform Module | Status |
-|---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
-| Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
-| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
-| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |
-
+| AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](flatcar-linux/aws.md) | stable |
+| Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](flatcar-linux/azure.md) | alpha |
+| Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](flatcar-linux/bare-metal.md) | stable |
+| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](flatcar-linux/digitalocean.md) | beta |
+| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](flatcar-linux/google-cloud.md) | beta |

 ## Documentation

 * Architecture [concepts](architecture/concepts.md) and [operating-systems](architecture/operating-systems.md)
 * Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Azure](fedora-coreos/azure.md), [Bare-Metal](fedora-coreos/bare-metal.md), [DigitalOcean](fedora-coreos/digitalocean.md), and [Google Cloud](fedora-coreos/google-cloud.md)
-* Flatcar Linux tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [DigitalOcean](cl/digital-ocean.md), and [Google Cloud](cl/google-cloud.md)
+* Flatcar Linux tutorials for [AWS](flatcar-linux/aws.md), [Azure](flatcar-linux/azure.md), [Bare-Metal](flatcar-linux/bare-metal.md), [DigitalOcean](flatcar-linux/digitalocean.md), and [Google Cloud](flatcar-linux/google-cloud.md)

 ## Example

@ -64,7 +53,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.18.4"

  # Google Cloud
  cluster_name  = "yavin"
@ -102,9 +91,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.4
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.4
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.4
 ```

 List the pods.
--- a/docs/topics/faq.md
+++ b/docs/topics/faq.md
@ -6,15 +6,6 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 Formats rise and evolve. Typhoon may choose to adapt the format over time (with lots of forewarning). However, the authors' have built several Kubernetes "distros" before and learned from mistakes - Terraform modules are the right format for now.

-## Operating Systems
-
-Typhoon supports Container Linux and the Flatcar Linux derivative. These operating systems were chosen because they offer:
-
-* Minimalism and focus on clustered operation
-* Automated and atomic operating system upgrades
-* Declarative and immutable configuration
-* Optimization for containerized applications
-
 ## Get Help

 Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar

 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.8.6"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.18.4"
  ...
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.4"
  ...
 }
 ```
@ -78,7 +78,7 @@ $ terraform apply
 Apply complete! Resources: 0 added, 0 changed, 55 destroyed.
 ```

-Re-provision a new cluster by following the bare-metal [tutorial](../cl/bare-metal.md#cluster).
+Re-provision a new cluster by following the bare-metal [tutorial](../fedora-coreos/bare-metal.md#cluster).

 ### Cloud

@ -127,84 +127,16 @@ Typhoon supports multi-controller clusters, so it is possible to upgrade a clust
 !!! warning
    Typhoon does not support or document node replacement as an upgrade strategy. It limits Typhoon's ability to make infrastructure and architectural changes between tagged releases.

-### Terraform Plugins Directory
-
-Use the Terraform 3rd-party [plugin directory](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) `~/.terraform.d/plugins` to keep versioned copies of the `terraform-provider-ct` and `terraform-provider-matchbox` plugins. The plugin directory replaces the `~/.terraformrc` file to allow 3rd party plugins to be defined and versioned independently (rather than globally).
-
-```
-# ~/.terraformrc (DEPRECATED)
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-  matchbox = "/usr/local/bin/terraform-provider-matchbox"
-}
-```
-
-Migrate to using the Terraform plugin directory. Move `~/.terraformrc` to a backup location.
-
-```
-mv ~/.terraformrc ~/.terraform-backup
-```
-
-Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`. Download the **same version** of `terraform-provider-ct` you were using with `~/.terraformrc`, updating only be done as a followup and is **only** safe for v1.12.2+ clusters!
-
-```sh
-wget https://github.com/poseidon/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.2.1
-```
-
-If you use bare-metal, add the [terraform-provider-matchbox](https://github.com/poseidon/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the versioned name.
-
-```sh
-wget https://github.com/poseidon/terraform-provider-matchbox/releases/download/v0.2.3/terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
-tar xzf terraform-provider-matchbox-v0.2.3-linux-amd64.tar.gz
-mv terraform-provider-matchbox-v0.2.3-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.3
-```
-
-Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
-
-```
-$ tree ~/.terraform.d/
-/home/user/.terraform.d/
-└── plugins
-    ├── terraform-provider-ct_v0.2.1
-    └── terraform-provider-matchbox_v0.2.3
-```
-
-In each Terraform working directory, set the version of each provider.
-
-```
-# providers.tf
-
-provider "matchbox" {
-  version = "0.2.3"
-  ...
-}
-
-provider "ct" {
-  version = "0.2.1"
-}
-```
-
-Run `terraform init` to ensure plugin version requirements are met. Verify `terraform plan` does not produce a diff, since the plugin versions should be the same as previously.
-
-```
-$ terraform init
-$ terraform plan
-```
-
 ### Upgrade terraform-provider-ct

-The [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin parses, validates, and converts Container Linux Configs into Ignition user-data for provisioning instances. Previously, updating the plugin re-provisioned controller nodes and was destructive to clusters. With Typhoon v1.12.2+, the plugin can be updated in-place and on apply, only workers will be replaced.
-
-First, [migrate](#terraform-plugins-directory) to the Terraform 3rd-party plugin directory to allow 3rd-party plugins to be defined and versioned independently (rather than globally).
+The [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin parses, validates, and converts Container Linux Configs into Ignition user-data for provisioning instances. Since Typhoon v1.12.2+, the plugin can be updated in-place so that on apply, only workers will be replaced.

 Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
-wget https://github.com/poseidon/terraform-provider-ct/releases/download/v0.3.1/terraform-provider-ct-v0.3.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.3.1-linux-amd64.tar.gz
-mv terraform-provider-ct-v0.3.1-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.1
+wget https://github.com/poseidon/terraform-provider-ct/releases/download/v0.5.0/terraform-provider-ct-v0.5.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.5.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.5.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.5.0
 ```

 Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
@ -215,8 +147,8 @@ $ tree ~/.terraform.d/
 └── plugins
    ├── terraform-provider-ct_v0.2.1
    ├── terraform-provider-ct_v0.3.0
-    ├── terraform-provider-ct_v0.3.1
-    └── terraform-provider-matchbox_v0.2.3
+    ├── terraform-provider-ct_v0.5.0
+    └── terraform-provider-matchbox_v0.3.0
 ```


@ -225,7 +157,7 @@ Update the version of the `ct` plugin in each Terraform working directory. Typho
 ```
 # providers.tf
 provider "ct" {
-  version = "0.3.0"
+  version = "0.5.0"
 }
 ```

@ -279,153 +211,9 @@ Typhoon modules have been adapted for Terraform v0.12. Provider plugins requirem

 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.18.2 - ?       | v0.12.x             |
-| v1.10.3 - v1.18.2 | v0.11.x             |
+| v1.15.0 - ?       | v0.12.x             |
+| v1.10.3 - v1.15.0 | v0.11.x             |
 | v1.9.2 - v1.10.2  | v0.10.4+ or v0.11.x |
 | v1.7.3 - v1.9.1   | v0.10.x             |
 | v1.6.4 - v1.7.2   | v0.9.x              |

-### New users
-
-New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.18.2+ without issue.
-
-### Existing users
-
-Migrate from Terraform v0.11 to v0.12 either **in-place** (easier, riskier) or by **moving resources** (safer, tedious).
-
-Install [Terraform](https://www.terraform.io/downloads.html) v0.12.x on your system alongside Terraform v0.11.x.
-
-```shell
-sudo ln -sf ~/Downloads/terraform-0.12.0/terraform /usr/local/bin/terraform12
-```
-
-!!! note
-    For example, `terraform` may refer Terraform v0.11.14, while `terraform12` is symlinked to Terraform v0.12.1. Once migration is complete, Terraform v0.11.x can be deleted and `terraform12` renamed.
-
-#### In-place
-
-For existing Typhoon v1.14.2 or v1.14.3 clusters, edit the Typhoon `ref` to first SHA that introduced Terraform v0.12 support (`3276bf587850218b8f967978a4bf2b05d5f440a2`). The aim is to minimize the diff and convert to using Terraform v0.12.x. For example:
-
-```tf
- module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.14.3"
-+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=3276bf587850218b8f967978a4bf2b05d5f440a2"
-   ...
-```
-
-With Terraform v0.12, Typhoon clusters no longer require the `providers` block (unless you actually need to pass an [aliased provider](https://www.terraform.io/docs/configuration/providers.html#alias-multiple-provider-instances)). A regression in Terraform v0.11 made it neccessary to explicitly pass aliased providers in order for Typhoon to continue to enforce constraints (see [terraform#16824](https://github.com/hashicorp/terraform/issues/16824)). Terraform v0.12 resolves this issue.
-
-```tf
- module "mercury" {
-   source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=3276bf587850218b8f967978a4bf2b05d5f440a2"
-
-  providers = {
-    local = "local.default"
-    null = "null.default"
-    template = "template.default"
-    tls = "tls.default"
-  }
-```
-
-Provider constrains ensure suitable plugin versions are used. Install new versions of `terraform-provider-ct` (v0.3.2+) and `terraform-provider-matchbox` (bare-metal only, v0.3.0+) according to the [changelog](https://github.com/poseidon/typhoon/blob/master/CHANGES.md#v1144) or tutorial docs. The `local`, `null`, `template`, and `tls` blocks in `providers.tf` are no longer needed.
-
-```tf
- provider "matchbox" {
-  version     = "0.2.3"
-+  version     = "0.3.0"
-   endpoint    = "matchbox.example.com:8081"
-   client_cert = "${file("~/.config/matchbox/client.crt")}"
-   client_key  = "${file("~/.config/matchbox/client.key")}"
- }
-
- provider "ct" {
-  version = "0.3.2"
-+  version = "0.3.3"
- }
-
-provider "local" {
-  version = "~> 1.0"
-  alias = "default"
-}
-
-provider "null" {
-  version = "~> 1.0"
-  alias = "default"
-}
-
-provider "template" {
-  version = "~> 1.0"
-  alias = "default"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-  alias = "default"
-}
-```
-
-Within the Terraform config directory (i.e. working directory), initialize to fetch suitable provider plugins.
-
-```shell
-terraform12 init  # using Terraform v0.12 binary, not v0.11
-```
-
-Use the Terraform v0.12 upgrade subcommand to convert v0.11 syntax to v0.12. This _will_ edit resource definitions in `*.tf` files in the working directory. Start from a clean version control state. Inspect the changes. Resolve any "TODO" items.
-
-```shell
-terraform12 0.12upgrade
-git diff
-```
-
-Finally, plan.
-
-```shell
-terraform12 plan
-```
-
-Verify no changes are proposed and commit changes to version control. You've migrated to Terraform v0.12! Repeat for other config directories. Use the Terraform v0.12 binary going forward.
-
-!!! note
-    It is known that plan may propose re-creating `template_dir` resources. This is harmless.
-
-!!! error
-    If plan produced errors, seek to address them (they may be in non-Typhoon resources). If plan proposed a diff, you'll need to evaluate whether that's expected and safe to apply. In-place edits between Typhoon releases aren't supported (favoring blue/green replacement). The larger the version skew, the greater the risk. Use good judgement. If in doubt, abandon the generated changes, delete `.terraform` as [suggested](https://www.terraform.io/upgrade-guides/0-12.html#upgrading-to-terraform-0-12), and try the move resources approach.
-
-#### Moving Resources
-
-Alternately, continue maintaining existing clusters using Terraform v0.11.x and existing Terraform configuration directory(ies). Create new Terraform directory(ies) and move resources there to be managed with Terraform v0.12. This approach allows resources to be migrated incrementally and ensures existing resources can always be managed (e.g. emergency patches).
-
-Create a new Terraform [config directory](/architecture/concepts/#organize) for *new* resources.
-
-```shell
-mkdir infra2
-tree .
-├── infraA  <- existing Terraform v0.11.x configs
-└── infraB  <- new Terraform v0.12.x configs
-```
-
-Define Typhoon clusters in the new config directory using Terraform v0.12 syntax. Follow the Typhoon v1.15.0+ docs (e.g. use `terraform12` in the `infraB` dir). See [AWS](/cl/aws), [Azure](/cl/azure), [Bare-Metal](/cl/bare-metal), [Digital Ocean](/cl/digital-ocean), or [Google-Cloud](/cl/google-cloud)) to create new clusters. Follow the usual [upgrade](/topics/maintenance/#upgrades) process to apply workloads and shift traffic. Later, switch back to the old config directory and deprovision clusters with Terraform v0.11.
-
-```shell
-terraform12 init
-terraform12 plan
-terraform12 apply
-```
-
-Your Terraform configuration directory likely defines resources other than just Typhoon modules (e.g. application DNS records, firewall rules, etc.). While such migrations are outside Typhoon's scope, you'll probably want to move existing resource definitions into your new Terraform configuration directory. Use Terraform v0.12 to import the resource into the state associated with the new config directory (to avoid trying to recreate a resource that exists). Then with Terraform v0.11 in the old directory, remove the resource from the state (to avoid trying to delete the resource). Verify neither `plan` produces a diff.
-
-```sh
-# move google_dns_record_set.some-app from infraA to infraB
-cd infraA
-terraform state list
-terraform state show google_dns_record_set.some-app
-
-cd ../infraB
-terraform12 import google_dns_record_set.some-app SOMEID
-terraform12 plan
-
-cd ../infraA
-terraform state rm google_dns_record_set.some-app
-terraform plan
-```
-
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Dalton Hubble	37f00a3882	Reduce Calcio MTU on Fedora CoreOS Azure * Change the Calico VXLAN interface for MTU from 1450 to 1410 * VXLAN on Azure should support MTU 1450. However, there is history where performance measures have shown that 1410 is needed to have expected performance. Flatcar Linux has the same MTU 1410 override and note * FCOS 31.20200323.3.2 was known to perform fine with 1450, but now in 31.20200517.3.0 the right value seems to be 1410	2020-06-19 00:24:56 -07:00
Dalton Hubble	4cfafeaa07	Fix Kubelet starting before hostname set on FCOS AWS * Fedora CoreOS `kubelet.service` can start before the hostname is set. Kubelet reads the hostname to determine the node name to register. If the hostname was read as localhost, Kubelet will continue trying to register as localhost (problem) * This race manifests as a node that appears NotReady, the Kubelet is trying to register as localhost, while the host itself (by then) has an AWS provided hostname. Restarting kubelet.service is a manual fix so Kubelet re-reads the hostname * This race could only be shown on AWS, not on Google Cloud or Azure despite attempts. Bare-metal and DigitalOcean differ and use hostname-override (e.g. afterburn) so they're not affected * Wait for nodes to have a non-localhost hostname in the oneshot that awaits /etc/resolve.conf. Typhoon has no valid cases for a node hostname being localhost (not even single-node clusters) Related Openshift: https://github.com/openshift/machine-config-operator/pull/1813 Close https://github.com/poseidon/typhoon/issues/765	2020-06-19 00:19:54 -07:00
Dalton Hubble	90e23f5822	Rename controller node label and NoSchedule taint * Remove node label `node.kubernetes.io/master` from controller nodes * Use `node.kubernetes.io/controller` (present since v1.9.5, [#160](https://github.com/poseidon/typhoon/pull/160)) to node select controllers * Rename controller NoSchedule taint from `node-role.kubernetes.io/master` to `node-role.kubernetes.io/controller` * Tolerate the new taint name for workloads that may run on controller nodes and stop tolerating `node-role.kubernetes.io/master` taint	2020-06-19 00:12:13 -07:00
Dalton Hubble	6234147948	Update recommended Terraform provider versions * Sync Terraform provider plugin versions with those used internally	2020-06-18 01:03:37 -07:00
Dalton Hubble	c25c59058c	Update Kubernetes from v1.18.3 to v1.18.4 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1184	2020-06-17 19:53:19 -07:00
Dalton Hubble	bc9b808d44	Update nginx-ingress from v0.32.0 to v0.33.0 * https://github.com/kubernetes/ingress-nginx/releases/tag/controller-0.33.0	2020-06-16 18:44:40 -07:00
Dalton Hubble	4b0203fdb2	Fix typo in DigitalOcean docs title	2020-06-16 18:33:56 -07:00
Dalton Hubble	331566e1f7	Update mkdocs packages for website	2020-06-16 18:20:19 -07:00
Dalton Hubble	04520e447c	Update node-exporter from v1.0.0 to v1.0.1 * https://github.com/prometheus/node_exporter/releases/tag/v1.0.1	2020-06-16 17:57:09 -07:00
Dalton Hubble	413585681b	Remove unused Kubelet lock-file and exit-on-lock-contention * Kubelet `--lock-file` and `--exit-on-lock-contention` date back to usage of bootkube and at one point running Kubelet in a "self-hosted" style whereby an on-host Kubelet (rkt) started pods, but then a Kubelet DaemonSet was scheduled and able to take over (hence self-hosted). `lock-file` and `exit-on-lock-contention` flags supported this pivot. The pattern has been out of favor (in bootkube too) for years because of dueling Kubelet complexity * Typhoon runs Kubelet as a container via an on-host systemd unit using podman (Fedora CoreOS) or rkt (Flatcar Linux). In fact, Typhoon no longer uses bootkube or control plane pivot (let alone Kubelet pivot) and uses static pods since v1.16.0 * https://github.com/poseidon/typhoon/pull/536	2020-06-12 00:06:41 -07:00
Dalton Hubble	96711d7f17	Remove unused Kubelet cert / key Terraform state * Generated Kubelet TLS certificate and key are not longer used or distributed to machines since Kubelet TLS bootstrap is used instead. Remove the certificate and key from state	2020-06-11 21:24:36 -07:00
Dalton Hubble	c9059d3fe9	Update Prometheus from v2.19.0-rc.0 to v2.19.0 * https://github.com/prometheus/prometheus/releases/tag/v2.19.0	2020-06-09 23:05:03 -07:00
Dalton Hubble	a287920169	Use strict mode for Container Linux Configs * Enable terraform-provider-ct `strict` mode for parsing Container Linux Configs and snippets * Fix Container Linux Config systemd unit syntax `enable` (old) to `enabled` * Align with Fedora CoreOS which uses strict mode already	2020-06-09 23:00:36 -07:00
Dalton Hubble	8dc170b9d9	Update security disclosure contact email * Use security@psdn.io across github.com/poseidon projects	2020-06-08 12:37:09 -07:00
Dalton Hubble	aed1a5f33d	Fix Fedora CoreOS docs for selecting a stream * Fedora CoreOS image `os_stream` stable, testing, and next have been configurable since v1.18.3 * Remove mention of outdated `os_image` variable	2020-06-08 12:25:57 -07:00
Dalton Hubble	31d02b0221	Update Prometheus from v2.18.1 to v2.19.0-rc.0 * https://github.com/prometheus/prometheus/releases/tag/v2.19.0-rc.0	2020-06-05 00:16:45 -07:00
Dalton Hubble	8f875f80f5	Update Grafana from v7.0.1 to v7.0.3 * https://github.com/grafana/grafana/releases/tag/v7.0.2 * https://github.com/grafana/grafana/releases/tag/v7.0.3	2020-06-03 12:31:58 -07:00
Dalton Hubble	16c0b9152b	Update kube-state-metrics from v1.9.6 to v1.9.7 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.7	2020-06-03 11:35:10 -07:00
Dalton Hubble	99dbce67a3	Tweak minor style elements of issue templates	2020-05-31 16:19:33 -07:00
Dalton Hubble	20bfd69780	Change Kubelet container image publishing * Build Kubelet container images internally and publish to Quay and Dockerhub (new) as an alternative in case of registry outage or breach * Use our infra to provide single and multi-arch (default) Kublet images for possible future use * Docs: Show how to use alternative Kubelet images via snippets and a systemd dropin (builds on #737) Changes: * Update docs with changes to Kubelet image building * If you prefer to trust images built by Quay/Dockerhub, automated image builds are still available with unique tags (albeit with some limitations): * Quay automated builds are tagged `build-{short_sha}` (limit: only amd64) * Dockerhub automated builts are tagged `build-{tag}` and `build-master` (limit: only amd64, no shas) Links: * Kubelet: https://github.com/poseidon/kubelet * Docs: https://typhoon.psdn.io/topics/security/#container-images * Registries: * quay.io/poseidon/kubelet * docker.io/psdn/kubelet	2020-05-30 23:34:23 -07:00
Dalton Hubble	ba44408b76	Update Calico from v3.14.0 to v3.14.1 * https://docs.projectcalico.org/v3.14/release-notes/	2020-05-30 22:08:37 -07:00
Dalton Hubble	455175d9e6	Update the fallback issue template * Even "blank" issues need to fill out the fallback template	2020-05-28 00:06:59 -07:00
Dalton Hubble	d45804b1f6	Update Github issue template to use drop-downs (#747 ) * Create a stricter bug report template * Highlight topics that are not accepted in issues: operation, support, debugging, advice, or Kubernetes concepts * Add a section to strongly suggest bug reports link a PR or describe a solution. This may be able to weed out topics that aren't focused bug reports	2020-05-27 23:49:25 -07:00
Dalton Hubble	907a96916f	Update mkdocs-material from v5.2.0 to v5.2.2 * https://github.com/squidfunk/mkdocs-material/releases/tag/5.2.2	2020-05-27 21:49:40 -07:00
Dalton Hubble	187bb17d39	Update Grafana from v7.0.0 to v7.0.1 * https://github.com/grafana/grafana/releases/tag/v7.0.1	2020-05-27 21:35:24 -07:00
Dalton Hubble	abc31c3711	Update node-exporter from v1.0.0-rc.1 to v1.0.0 * https://github.com/prometheus/node_exporter/releases/tag/v1.0.0	2020-05-27 21:33:03 -07:00
Dalton Hubble	283e14f3e0	Update recommended Terraform provider versions * Sync Terraform provider plugin versions to those actively used internally * Fix terraform fmt	2020-05-22 01:12:53 -07:00
Dalton Hubble	e72f916c8d	Update etcd from v3.4.8 to v3.4.9 * https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.4.md#v349-2020-05-20	2020-05-22 00:52:20 -07:00
Dalton Hubble	c52f9f8d08	Upgrade docs packages and refresh content * Promote DigitalOcean from alpha to beta for Fedora CoreOS and Flatcar Linux * Upgrade mkdocs-material and PyPI packages for docs * Replace docs mentions of Container Linux with Flatcar Linux and move docs/cl to docs/flatcar-linux * Deprecate CoreOS Container Linux support. Its still usable for some time, but start removing docs	2020-05-20 23:31:26 -07:00
Dalton Hubble	ecae6679ff	Update Kubernetes from v1.18.2 to v1.18.3 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md	2020-05-20 20:37:39 -07:00
Dalton Hubble	4760543356	Set Kubelet image via kubelet.service KUBELET_IMAGE * Write the systemd kubelet.service to use `KUBELET_IMAGE` as the Kubelet. This provides a nice way to use systemd dropins to temporarily override the image (e.g. during a registry outage) Note: Only Typhoon Kubelet images and registries are supported.	2020-05-19 22:39:53 -07:00
Dalton Hubble	09eb208b4e	Fix Fedora CoreOS on GCP proposing controller recreate * With Fedora CoreOS image stream support (#727), the latest resolved image will change over the lifecycle of a cluster. * Fix issue where an image diff proposed replacing a Fedora CoreOS controller on GCP, introduced in #727 (unreleased) * Also ignore image diffs to the GCP managed instance group of workers. This aligns with worker AMI diffs being ignored on AWS and similar on Azure, since workers update themselves. Background: * Controller nodes should strictly not be recreated by Terraform, they are stateful (etcd) and should not be replaced * Across cloud platforms, OS image diffs are ignored since both Flatcar Linux and Fedora CoreOS nodes update themselves. For workers, user-data or disk size diffs (where relevant) are allowed to recreate workers templates/configs since these are considered to be user-initiated declarations that a reprovision should be done	2020-05-19 21:41:51 -07:00
Dalton Hubble	8d024d22ad	Update etcd from v3.4.7 to v3.4.8 * https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.4.md#v348-2020-05-18	2020-05-18 23:50:46 -07:00
Dalton Hubble	3bdddc452c	Update Grafana from v7.0.0-beta2 to v7.0.0 * https://grafana.com/docs/grafana/latest/guides/whats-new-in-v7-0/	2020-05-18 23:42:32 -07:00
Dalton Hubble	ff4187a1fb	Use new Azure subnet to set address_prefixes list * Update Azure subnet `address_prefix` to `azure_prefixes` list * Fix warning that `address_prefix` is deprecated * Require `terraform-provider-azurerm` v2.8.0+ (action required) Rel: https://github.com/terraform-providers/terraform-provider-azurerm/pull/6493	2020-05-18 23:35:47 -07:00
Dalton Hubble	2578be1f96	Rollback Grafana to v7.0.0-beta3, v7.0.0 image is missing * Grafana hasn't published the v7.0.0 image yet	2020-05-16 12:32:10 -07:00
Dalton Hubble	90edcd3d77	Update node-exporter from v1.0.0-rc.0 to v1.0.0-rc.1 * https://github.com/prometheus/node_exporter/releases/tag/v1.0.0-rc.1	2020-05-15 18:03:19 -07:00
Dalton Hubble	a927c7c790	Update kube-state-metrics from v1.9.5 to v1.9.6 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.6	2020-05-15 17:42:24 -07:00
Dalton Hubble	d952576d2f	Update Grafana from v7.0.0-beta3 to v7.0.0 * https://github.com/grafana/grafana/releases/tag/7.0.0	2020-05-15 17:38:59 -07:00
Dalton Hubble	70e389f37f	Restore use of Flatcar Linux Azure Marketplace image * Switch Flatcar Linux Azure to use the Marketplace image from Kinvolk (offer `flatcar-container-linux-free`) * Accepting Azure Marketplace terms is still neccessary, update docs to show accepting the free offer rather than BYOL * Upstream Flatcar: https://github.com/flatcar-linux/Flatcar/issues/82 * Typhoon: https://github.com/poseidon/typhoon/issues/703	2020-05-13 22:50:24 -07:00
Dalton Hubble	a18bd0a707	Highlight SELinux enforcing mode in features	2020-05-13 21:57:38 -07:00
Dalton Hubble	01905b00bc	Support Fedora CoreOS OS image streams on AWS * Add `os_stream` variable to set the stream to stable (default), testing, or next * Remove unused os_image variable on Fedora CoreOS AWS	2020-05-13 21:45:12 -07:00
Dalton Hubble	f4194cd57a	Update Grafana from v7.0.0-beta2 to v7.0.0-beta.3 * https://github.com/grafana/grafana/releases/tag/v7.0.0-beta3	2020-05-09 17:50:40 -07:00
Dalton Hubble	a2db4fa8c4	Update Calico from v3.13.3 to v3.14.0 * https://docs.projectcalico.org/v3.14/release-notes/	2020-05-09 16:05:30 -07:00
Dalton Hubble	358854e712	Fix Calico install-cni crash loop on Pod restarts * Set a consistent MCS level/range for Calico install-cni * Note: Rebooting a node was a workaround, because Kubelet relabels /etc/kubernetes(/cni/net.d) Background: * On SELinux enforcing systems, the Calico CNI install-cni container ran with default SELinux context and a random MCS pair. install-cni places CNI configs by first creating a temporary file and then moving them into place, which means the file MCS categories depend on the containers SELinux context. * calico-node Pod restarts creates a new install-cni container with a different MCS pair that cannot access the earlier written file (it places configs every time), causing the init container to error and calico-node to crash loop * https://github.com/projectcalico/cni-plugin/issues/874 ``` mv: inter-device move failed: '/calico.conf.tmp' to '/host/etc/cni/net.d/10-calico.conflist'; unable to remove target: Permission denied Failed to mv files. This may be caused by selinux configuration on the host, or something else. ``` Note, this isn't a host SELinux configuration issue. Related: * https://github.com/poseidon/terraform-render-bootstrap/pull/186	2020-05-09 16:01:44 -07:00
Dalton Hubble	b5dabcea31	Use Fedora CoreOS image streams on Google Cloud * Add `os_stream` variable to set a Fedora CoreOS stream to `stable` (default), `testing`, or `next` * Deprecate `os_image` variable. Remove docs about uploading Fedora CoreOS images manually, this is no longer needed * https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/ Rel: https://github.com/coreos/fedora-coreos-docs/pull/70	2020-05-08 01:23:12 -07:00
Dalton Hubble	3f0a5d2715	Update Grafana from v7.0.0-beta1 to v7.0.0-beta2 * https://github.com/grafana/grafana/releases/tag/v7.0.0-beta2	2020-05-07 23:04:44 -07:00
Dalton Hubble	33173c0206	Update Prometheus from v2.18.0 to v2.18.1 * https://github.com/prometheus/prometheus/releases/tag/v2.18.1	2020-05-07 22:59:11 -07:00
Dalton Hubble	70f30d9c07	Update Prometheus from v2.18.0-rc.1 to v2.18.0 * https://github.com/prometheus/prometheus/releases/tag/v2.18.0	2020-05-05 22:31:11 -07:00
Dalton Hubble	6afc1643d9	Update nginx-ingress from v0.30.0 to v0.32.0 * Add support for IngressClass and RBAC authorization * Since our nginx ingress controller example uses the flag `--ingress-class=public`, add an IngressClass to go along with it Rel: https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class	2020-05-03 23:24:19 -07:00
Dalton Hubble	e71e27e769	Update Prometheus from v2.17.2 to v2.18.0-rc.1 * https://github.com/prometheus/prometheus/releases/tag/v2.18.0-rc.1	2020-04-29 20:57:48 -07:00
Dalton Hubble	64035005d4	Update Grafana from v6.7.2 to v7.0.0-beta1 * https://github.com/grafana/grafana/releases/tag/v7.0.0-beta1	2020-04-29 20:53:30 -07:00
Ben Drucker	317416b316	Use Terraform element wrap-around for AWS controllers subnet_id (#714 ) * Fix Terraform plan error when controller_count exceeds available AWS zones (e.g. 5 controllers)	2020-04-29 20:41:08 -07:00
Dalton Hubble	2c1af917ec	Update recommended Terraform provider versions * Sync the Terraform provider plugin versions to those actively used and tested by the author * Fix terraform fmt	2020-04-28 19:57:50 -07:00
Dalton Hubble	4ac2d94999	Add Fedora CoreOS Azure docs to site navigation * Fix missing Fedora CoreOS Azure docs	2020-04-28 19:54:37 -07:00
Dalton Hubble	fd044ee117	Enable Kubelet TLS bootstrap and NodeRestriction * Enable bootstrap token authentication on kube-apiserver * Generate the bootstrap.kubernetes.io/token Secret that may be used as a bootstrap token * Generate a bootstrap kubeconfig (with a bootstrap token) to be securely distributed to nodes. Each Kubelet will use the bootstrap kubeconfig to authenticate to kube-apiserver as `system:bootstrappers` and send a node-unique CSR for kube-controller-manager to automatically approve to issue a Kubelet certificate and kubeconfig (expires in 72 hours) * Add ClusterRoleBinding for bootstrap token subjects (`system:bootstrappers`) to have the `system:node-bootstrapper` ClusterRole * Add ClusterRoleBinding for bootstrap token subjects (`system:bootstrappers`) to have the csr nodeclient ClusterRole * Add ClusterRoleBinding for bootstrap token subjects (`system:bootstrappers`) to have the csr selfnodeclient ClusterRole * Enable NodeRestriction admission controller to limit the scope of Node or Pod objects a Kubelet can modify to those of the node itself * Ability for a Kubelet to delete its Node object is retained as preemptible nodes or those in auto-scaling instance groups need to be able to remove themselves on shutdown. This need continues to have precedence over any risk of a node deleting itself maliciously Security notes: 1. Issued Kubelet certificates authenticate as user `system:node:NAME` and group `system:nodes` and are limited in their authorization to perform API operations by Node authorization and NodeRestriction admission. Previously, a Kubelet's authorization was broader. This is the primary security motivation. 2. The bootstrap kubeconfig credential has the same sensitivity as the previous generated TLS client-certificate kubeconfig. It must be distributed securely to nodes. Its compromise still allows an attacker to obtain a Kubelet kubeconfig 3. Bootstrapping Kubelet kubeconfig's with a limited lifetime offers a slight security improvement. * An attacker who obtains the kubeconfig can likely obtain the bootstrap kubeconfig as well, to obtain the ability to renew their access * A compromised bootstrap kubeconfig could plausibly be handled by replacing the bootstrap token Secret, distributing the token to new nodes, and expiration. Whereas a compromised TLS-client certificate kubeconfig can't be revoked (no CRL). However, replacing a bootstrap token can be impractical in real cluster environments, so the limited lifetime is mostly a theoretical benefit. * Cluster CSR objects are visible via kubectl which is nice 4. Bootstrapping node-unique Kubelet kubeconfigs means Kubelet clients have more identity information, which can improve the utility of audits and future features Rel: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/ Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/185	2020-04-28 19:35:33 -07:00
Dalton Hubble	38a6bddd06	Update Calico from v3.13.1 to v3.13.3 * https://docs.projectcalico.org/v3.13/release-notes/	2020-04-23 23:58:02 -07:00
Dalton Hubble	d8966afdda	Remove extraneous sudo from layout asset unpacking	2020-04-22 20:28:01 -07:00
Dalton Hubble	84ed0a31c3	Update Prometheus from v2.17.1 to v2.17.2 * https://github.com/prometheus/prometheus/releases/tag/v2.17.2	2020-04-20 18:09:24 -07:00