Bump terraform-render-bootstrap

* https://github.com/prometheus/prometheus/releases/tag/v2.28.0
* https://github.com/grafana/grafana/releases/tag/v8.0.3

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: [poseidon]

.github/dependabot.yaml

@@ -3,7 +3,7 @@ updates:
 - package-ecosystem: pip
   directory: "/"
   schedule:
-    interval: daily
+    interval: weekly
   pull-request-branch-name:
     separator: "-"
   open-pull-requests-limit: 3

CHANGES.md

@@ -4,6 +4,110 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.21.2
+
+* Kubernetes [v1.21.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1212)
+* Add Terraform v1.0.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x, v0.14.4+, and v0.15.x
+* Update CoreDNS from v1.8.0 to [v1.8.4]([#1006](https://github.com/poseidon/typhoon/pull/1006))
+* Update Cilium from v1.9.6 to [v1.10.1](https://github.com/cilium/cilium/releases/tag/v1.10.1)
+* Update Calico from v3.19.0 to [v3.19.1](https://github.com/projectcalico/calico/releases/tag/v3.19.1)
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0 to [v2.1.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.0)
+* Update Prometheus from v2.27.0 to [v2.28.0](https://github.com/prometheus/prometheus/releases/tag/v2.28.0)
+* Update Grafana from v7.5.6 to [v8.0.3](https://github.com/grafana/grafana/releases/tag/v8.0.3)
+* Update nginx-ingress from v0.46.0 to [v0.47.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.47.0)
+
+### Fedora CoreOS
+
+#### AWS
+
+* Extend experimental Fedora CoreOS arm64 support with Cilium
+  * CNI provider may now be `flannel` or `cilium` (new)
+
+#### Bare-Metal
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+#### DigitalOcean
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881))
+
+## v1.21.1
+
+* Kubernetes [v1.21.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211)
+* Add Terraform v0.15.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x and v0.14.4+
+* Update etcd from v3.4.15 to [v3.4.16](https://github.com/etcd-io/etcd/releases/tag/v3.4.16)
+* Update Cilium from v1.9.5 to [v1.9.6](https://github.com/cilium/cilium/releases/tag/v1.9.6)
+* Update Calico from v3.18.1 to [v3.19.0](https://github.com/projectcalico/calico/releases/tag/v3.19.0)
+
+### AWS
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Azure
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Google Cloud
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Fedora CoreOS
+
+* Update Kubelet mounts for cgroups v2 ([#978](https://github.com/poseidon/typhoon/pull/978))
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0-rc.1 to [v2.0.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0)
+* Update Prometheus from v2.25.2 to [v2.27.0](https://github.com/prometheus/prometheus/releases/tag/v2.27.0)
+* Update Grafana from v7.5.3 to [v7.5.6](https://github.com/grafana/grafana/releases/tag/v7.5.6)
+* Update nginx-ingress from v0.45.0 to [v0.46.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.46.0)
+
+## v1.21.0
+
+* Kubernetes [v1.21.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1210)
+  * Enable `tokencleaner` controller ([#969](https://github.com/poseidon/typhoon/pull/969))
+  * Enable `kube-scheduler` and `kube-controller-manager` separate authn/z kubeconfig
+  * Change CNI config location from /etc/kubernetes/cni/net.d to /etc/cni/net.d ([#965](https://github.com/poseidon/typhoon/pull/965))
+  * Change `kube-controller-manager` to mount `/var/lib/kubelet/volumeplugins` directly
+  * Remove unused `cloud-provider` flags
+* Update Fedora CoreOS Config version from v1.1.0 to v1.2.0 ([#970](https://github.com/poseidon/typhoon/pull/970))
+  * Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.8+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.2.0
+
+### AWS
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Azure
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+* Remove deprecated `azurerm_lb_backend_address_pool` field `resource_group_name` ([#972](https://github.com/poseidon/typhoon/pull/972))
+
+### Google Cloud
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Addons
+
+* Update nginx-ingress from v0.44.0 to [v0.45.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.45.0)
+* Update kube-state-metrics from v2.0.0-rc.0 to [v2.0.0-rc.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-rc.1)
+* Update Grafana from v7.4.5 to [v7.5.3](https://github.com/grafana/grafana/releases/tag/v7.5.3)
+
 ## v1.20.5
 
 * Kubernetes [v1.20.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1205)

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@@ -31,6 +31,10 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | stable |
 
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora CoreOS (ARM64) | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | alpha |
+
 Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).
 
 | Platform      | Operating System | Terraform Module | Status |
@@ -54,7 +58,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -93,9 +97,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.5
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.5
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.5
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.2
 ```
 
 List the pods.
@@ -126,7 +130,7 @@ Typhoon is strict about minimalism, maturity, and scope. These are not in scope:
 
 ## Help
 
-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
+Schedule a meeting via [Github Sponsors](https://github.com/sponsors/poseidon?frequency=one-time) to discuss your use case.
 
 ## Motivation
 

addons/grafana/deployment.yaml

@@ -24,7 +24,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: grafana
-          image: docker.io/grafana/grafana:7.4.5
+          image: docker.io/grafana/grafana:8.0.3
           env:
             - name: GF_PATHS_CONFIG
               value: "/etc/grafana/custom.ini"

addons/nginx-ingress/aws/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/azure/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/prometheus/deployment.yaml

@@ -21,7 +21,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.25.2
+          image: quay.io/prometheus/prometheus:v2.28.0
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -25,7 +25,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-rc.0
+        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.0
         ports:
           - name: metrics
             containerPort: 8080

aws/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15${etcd_arch}
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16${etcd_arch}
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -54,9 +54,9 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -67,12 +67,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -90,7 +90,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -123,7 +122,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.21.2
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

aws/fedora-coreos/kubernetes/variables.tf

@@ -55,7 +55,7 @@ variable "os_stream" {
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
@@ -176,4 +176,3 @@ variable "daemonset_tolerations" {
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
   default     = []
 }
-

aws/fedora-coreos/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: docker.service
@@ -27,9 +27,9 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -40,12 +40,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -63,7 +63,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -91,7 +90,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

aws/fedora-coreos/kubernetes/workers/variables.tf

@@ -48,7 +48,7 @@ variable "os_stream" {
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {

aws/fedora-coreos/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

aws/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -12,5 +12,6 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }
 

aws/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -57,9 +57,9 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -70,6 +70,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -90,7 +91,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -119,7 +119,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

aws/flatcar-linux/kubernetes/variables.tf

@@ -55,7 +55,7 @@ variable "os_image" {
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
@@ -160,3 +160,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

aws/flatcar-linux/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

aws/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -29,9 +29,9 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -45,6 +45,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -65,7 +66,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -73,6 +73,9 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
           --read-only-port=0 \
@@ -91,7 +94,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

aws/flatcar-linux/kubernetes/workers/variables.tf

@@ -48,7 +48,7 @@ variable "os_image" {
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
@@ -113,3 +113,9 @@ variable "node_labels" {
   description = "List of initial node labels"
   default     = []
 }
+
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}

aws/flatcar-linux/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

aws/flatcar-linux/kubernetes/workers/workers.tf

@@ -86,6 +86,7 @@ data "template_file" "worker-config" {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

azure/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -18,6 +18,7 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 
   # Fedora CoreOS
   trusted_certs_dir = "/etc/pki/tls/certs"

azure/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -51,8 +51,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -63,12 +63,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -86,7 +86,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -118,7 +117,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.21.2
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

azure/fedora-coreos/kubernetes/lb.tf

@@ -112,16 +112,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {
 
 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "controller"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "worker"
   loadbalancer_id = azurerm_lb.cluster.id
 }

azure/fedora-coreos/kubernetes/variables.tf

@@ -54,7 +54,7 @@ variable "os_image" {
 variable "disk_size" {
   type        = number
   description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "worker_priority" {
@@ -135,3 +135,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

azure/fedora-coreos/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: docker.service
@@ -24,8 +24,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -36,12 +36,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -59,7 +59,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -67,6 +66,9 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
           --rotate-certificates \
@@ -83,7 +85,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

azure/fedora-coreos/kubernetes/workers/variables.tf

@@ -88,6 +88,12 @@ variable "node_labels" {
   default     = []
 }
 
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported
 
 variable "cluster_domain_suffix" {

azure/fedora-coreos/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

azure/fedora-coreos/kubernetes/workers/workers.tf

@@ -87,6 +87,7 @@ data "template_file" "worker-config" {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

azure/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -18,5 +18,6 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }
 

azure/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -55,8 +55,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -67,6 +67,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -87,7 +88,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -115,7 +115,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

azure/flatcar-linux/kubernetes/lb.tf

@@ -112,16 +112,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {
 
 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "controller"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "worker"
   loadbalancer_id = azurerm_lb.cluster.id
 }

azure/flatcar-linux/kubernetes/variables.tf

@@ -60,7 +60,7 @@ variable "os_image" {
 variable "disk_size" {
   type        = number
   description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "worker_priority" {
@@ -141,3 +141,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

azure/flatcar-linux/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

azure/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -27,8 +27,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -42,6 +42,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -62,7 +63,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -70,6 +70,9 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
           --rotate-certificates \
@@ -87,7 +90,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

azure/flatcar-linux/kubernetes/workers/variables.tf

@@ -94,6 +94,12 @@ variable "node_labels" {
   default     = []
 }
 
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported
 
 variable "cluster_domain_suffix" {

azure/flatcar-linux/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

azure/flatcar-linux/kubernetes/workers/workers.tf

@@ -105,6 +105,7 @@ data "template_file" "worker-config" {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

bare-metal/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -50,8 +50,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -62,12 +62,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -85,7 +85,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -120,7 +119,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \

bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: docker.service
@@ -23,8 +23,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -35,12 +35,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -58,7 +58,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \

bare-metal/fedora-coreos/kubernetes/ssh.tf

@@ -39,6 +39,7 @@ resource "null_resource" "copy-controller-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
       "sudo /opt/bootstrap/layout",
     ]
   }
@@ -70,6 +71,7 @@ resource "null_resource" "copy-worker-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
     ]
   }
 }

bare-metal/fedora-coreos/kubernetes/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
 
     matchbox = {

bare-metal/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -63,8 +63,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -75,6 +75,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -95,7 +96,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -124,7 +124,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

bare-metal/flatcar-linux/kubernetes/cl/worker.yaml

@@ -35,8 +35,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -50,6 +50,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -70,7 +71,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \

bare-metal/flatcar-linux/kubernetes/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
 
     matchbox = {

digital-ocean/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -52,9 +52,9 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -65,12 +65,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -88,7 +88,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -130,7 +129,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.21.2
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: docker.service
@@ -26,9 +26,9 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -39,12 +39,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -62,7 +62,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -93,7 +92,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

digital-ocean/fedora-coreos/kubernetes/ssh.tf

@@ -36,6 +36,7 @@ resource "null_resource" "copy-controller-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
       "sudo /opt/bootstrap/layout",
     ]
   }
@@ -60,6 +61,7 @@ resource "null_resource" "copy-worker-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
     ]
   }
 }
@@ -84,4 +86,3 @@ resource "null_resource" "bootstrap" {
     ]
   }
 }
-

digital-ocean/fedora-coreos/kubernetes/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
 
     digitalocean = {

digital-ocean/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -65,9 +65,9 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -78,6 +78,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -98,7 +99,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -127,7 +127,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml

@@ -37,9 +37,9 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -53,6 +53,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -73,7 +74,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -96,7 +96,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

digital-ocean/flatcar-linux/kubernetes/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
 
     digitalocean = {

docs/advanced/arm64.md

@@ -6,7 +6,7 @@
 Typhoon has experimental support for ARM64 with Fedora CoreOS on AWS. Full clusters can be created with ARM64 controller and worker nodes. Or worker pools of ARM64 nodes can be attached to an AMD64 cluster to create a hybrid/mixed architecture cluster.
 
 !!! note
-    Currently, CNI networking must be set to flannel.
+    Currently, CNI networking must be set to flannel or Cilium.
 
 ## AMIs
 
@@ -21,7 +21,7 @@ Create a cluster with ARM64 controller and worker nodes. Container workloads mus
 
 ```tf
 module "gravitas" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.19.4"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # AWS
   cluster_name = "gravitas"
@@ -33,7 +33,7 @@ module "gravitas" {
 
   # optional
   arch         = "arm64"
-  networking   = "flannel"
+  networking   = "cilium"
   worker_count = 2
   worker_price = "0.0168"
 
@@ -47,9 +47,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes.
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-12-178   Ready    <none>   101s   v1.19.4   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-18-93    Ready    <none>   102s   v1.19.4   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-90-10    Ready    <none>   104s   v1.19.4   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-12-178   Ready    <none>   101s   v1.21.2   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-18-93    Ready    <none>   102s   v1.21.2   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-90-10    Ready    <none>   104s   v1.21.2   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
 ```
 
 ## Hybrid
@@ -60,7 +60,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.19.4"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.21.2"
 
       # AWS
       cluster_name = "gravitas"
@@ -71,7 +71,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
       ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
 
       # optional
-      networking   = "flannel"
+      networking   = "cilium"
       worker_count = 2
       worker_price = "0.021"
 
@@ -83,7 +83,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.19.4"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.21.2"
 
       # AWS
       vpc_id          = module.gravitas.vpc_id
@@ -107,10 +107,10 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 
 ```
 $ kubectl get nodes -o wide
-NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-14-73    Ready    <none>   116s   v1.19.4   10.0.14.73    <none>        Fedora CoreOS 32.20201018.3.0     5.8.15-201.fc32.x86_64    docker://19.3.11
-ip-10-0-17-167   Ready    <none>   104s   v1.19.4   10.0.17.167   <none>        Fedora CoreOS 32.20201018.3.0     5.8.15-201.fc32.x86_64    docker://19.3.11
-ip-10-0-47-166   Ready    <none>   110s   v1.19.4   10.0.47.166   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-7-237    Ready    <none>   111s   v1.19.4   10.0.7.237    <none>        Fedora CoreOS 32.20201018.3.0     5.8.15-201.fc32.x86_64    docker://19.3.11
+NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION             CONTAINER-RUNTIME
+ip-10-0-1-81    Ready    <none>   4m28s   v1.21.2   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-17-86   Ready    <none>   4m28s   v1.21.2   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
+ip-10-0-21-45   Ready    <none>   4m28s   v1.21.2   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-40-36   Ready    <none>   4m22s   v1.21.2   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
 ```
 

docs/advanced/customization.md

@@ -37,7 +37,7 @@ For example, ensure an `/opt/hello` file is created with permissions 0644.
 ```yaml
 # custom-files
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 storage:
   files:
     - path: /opt/hello
@@ -185,7 +185,7 @@ To set an alternative etcd image or Kubelet image, use a snippet to set a system
     ```yaml
     # kubelet-image-override.yaml
     variant: fcos           <- remove for Flatcar Linux
-    version: 1.1.0          <- remove for Flatcar Linux
+    version: 1.2.0          <- remove for Flatcar Linux
     systemd:
       units:
         - name: kubelet.service
@@ -201,7 +201,7 @@ To set an alternative etcd image or Kubelet image, use a snippet to set a system
     ```yaml
     # etcd-image-override.yaml
     variant: fcos           <- remove for Flatcar Linux
-    version: 1.1.0          <- remove for Flatcar Linux
+    version: 1.2.0          <- remove for Flatcar Linux
     systemd:
       units:
         - name: etcd-member.service

docs/advanced/nodes.md

@@ -0,0 +1,134 @@
+# Nodes
+
+Typhoon clusters consist of controller node(s) and a (default) set of worker nodes.
+
+## Overview
+
+Typhoon nodes use the standard set of Kubernetes node labels.
+
+```yaml
+Labels: kubernetes.io/arch=amd64
+        kubernetes.io/hostname=node-name
+        kubernetes.io/os=linux
+```
+
+Controller node(s) are labeled to allow node selection (for rare components that run on controllers) and tainted to prevent ordinary workloads running on controllers.
+
+```yaml
+Labels: node.kubernetes.io/controller=true
+Taints: node-role.kubernetes.io/controller:NoSchedule
+```
+
+Worker nodes are labeled to allow node selection and untainted. Workloads will schedule on worker nodes by default, baring any contraindications.
+
+```yaml
+Labels: node.kubernetes.io/node=
+Taints: <none>
+```
+
+On auto-scaling cloud platforms, you may add [worker pools](/advanced/worker-pools/) with different groups of nodes with their own labels and taints. On platforms like bare-metal, with heterogeneous machines, you may manage node labels and taints per node.
+
+## Node Labels
+
+Add custom initial worker node labels to default workers or worker pool nodes to allow workloads to select among nodes that differ.
+
+=== "Cluster"
+
+    ```tf
+    module "yavin" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.2"
+
+      # Google Cloud
+      cluster_name  = "yavin"
+      region        = "us-central1"
+      dns_zone      = "example.com"
+      dns_zone_name = "example-zone"
+
+      # configuration
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count = 2
+      worker_node_labels = ["pool=default"]
+    }
+    ```
+
+=== "Worker Pool"
+
+    ```tf
+    module "yavin-pool" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.21.2"
+
+      # Google Cloud
+      cluster_name = "yavin"
+      region       = "europe-west2"
+      network      = module.yavin.network_name
+
+      # configuration
+      name               = "yavin-16x"
+      kubeconfig         = module.yavin.kubeconfig
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count = 1
+      machine_type = "n1-standard-16"
+      node_labels  = ["pool=big"]
+    }
+    ```
+
+In the example above, the two default workers would be labeled `pool: default` and the additional worker would be labeled `pool: big`.
+
+## Node Taints
+
+Add custom initial taints on worker pool nodes to indicate a node is unique and should only schedule workloads that explicitly tolerate a given taint key.
+
+!!! warning
+    Since taints prevent workloads scheduling onto a node, you must decide whether `kube-system` DaemonSets (e.g. flannel, Calico, Cilium) should tolerate your custom taint by setting `daemonset_tolerations`. If you don't list your custom taint(s), important components won't run on these nodes.
+
+=== "Cluster"
+
+    ```tf
+    module "yavin" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.2"
+
+      # Google Cloud
+      cluster_name  = "yavin"
+      region        = "us-central1"
+      dns_zone      = "example.com"
+      dns_zone_name = "example-zone"
+
+      # configuration
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count = 2
+      daemonset_tolerations = ["role"]
+    }
+    ```
+
+=== "Worker Pool"
+
+    ```tf
+    module "yavin-pool" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.21.2"
+
+      # Google Cloud
+      cluster_name = "yavin"
+      region       = "europe-west2"
+      network      = module.yavin.network_name
+
+      # configuration
+      name               = "yavin-16x"
+      kubeconfig         = module.yavin.kubeconfig
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count      = 1
+      accelerator_type  = "nvidia-tesla-p100"
+      accelerator_count = 1
+      node_taints       = ["role=gpu:NoSchedule"]
+    }
+    ```
+
+In the example above, the the additional worker would be tainted with `role=gpu:NoSchedule` to prevent workloads scheduling, but `kube-system` components like flannel, Calico, or Cilium would tolerate that custom taint to run there.
+

docs/advanced/worker-pools.md

@@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.20.5"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.21.2"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.20.5"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.21.2"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -99,6 +99,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | snippets | Fedora CoreOS or Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |
+| node_taints | List of initial node taints | [] | ["role=gpu:NoSchedule"] |
 
 Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/) or per-region and per-type [spot prices](https://aws.amazon.com/ec2/spot/pricing/).
 
@@ -110,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.20.5"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.21.2"
 
       # Azure
       region                  = module.ramius.region
@@ -136,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.20.5"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.21.2"
 
       # Azure
       region                  = module.ramius.region
@@ -194,6 +195,7 @@ The Azure internal `workers` module supports a number of [variables](https://git
 | snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |
+| node_taints | List of initial node taints | [] | ["role=gpu:NoSchedule"] |
 
 Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
 
@@ -205,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.20.5"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.21.2"
 
       # Google Cloud
       region       = "europe-west2"
@@ -229,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.20.5"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.21.2"
 
       # Google Cloud
       region       = "europe-west2"
@@ -260,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.20.5
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.20.5
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.20.5
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.20.5
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.20.5
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.21.2
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.21.2
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.21.2
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.21.2
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.21.2
 ```
 
 ### Variables
@@ -297,6 +299,7 @@ Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-z
 | snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |
+| node_taints | List of initial node taints | [] | ["role=gpu:NoSchedule"] |
 
 Check the list of valid [machine types](https://cloud.google.com/compute/docs/machine-types).
 

docs/architecture/operating-systems.md

@@ -46,7 +46,7 @@ Typhoon conventional directories.
 
 | Kubelet setting   | Host location                  |
 |-------------------|--------------------------------|
-| cni-conf-dir      | /etc/kubernetes/cni/net.d      |
+| cni-conf-dir      | /etc/cni/net.d                 |
 | pod-manifest-path | /etc/kubernetes/manifests      |
 | volume-plugin-dir | /var/lib/kubelet/volumeplugins |
 

docs/fedora-coreos/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on AWS with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -51,11 +51,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     aws = {
       source = "hashicorp/aws"
-      version = "3.33.0"
+      version = "3.39.0"
     }
   }
 }
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.20.5
-ip-10-0-26-65  Ready   <none>   10m  v1.20.5
-ip-10-0-41-21  Ready   <none>   10m  v1.20.5
+ip-10-0-3-155  Ready   <none>   10m  v1.21.2
+ip-10-0-26-65  Ready   <none>   10m  v1.21.2
+ip-10-0-41-21  Ready   <none>   10m  v1.21.2
 ```
 
 List the pods.
@@ -211,7 +211,7 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
-| disk_size | Size of the EBS volume in GB | 40 | 100 |
+| disk_size | Size of the EBS volume in GB | 30 | 100 |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |

docs/fedora-coreos/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on Azure with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -48,11 +48,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     azurerm = {
       source = "hashicorp/azurerm"
-      version = "2.52.0"
+      version = "2.58.0"
     }
   }
 }
@@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # Azure
   cluster_name   = "ramius"
@@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.20.5
-ramius-worker-000001  Ready   <none>  25m  v1.20.5
-ramius-worker-000002  Ready   <none>  24m  v1.20.5
+ramius-controller-0   Ready   <none>  24m  v1.21.2
+ramius-worker-000001  Ready   <none>  25m  v1.21.2
+ramius-worker-000002  Ready   <none>  24m  v1.21.2
 ```
 
 List the pods.
@@ -241,7 +241,7 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "Standard_B2s" | See below |
 | worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
-| disk_size | Size of the disk in GB | 40 | 100 |
+| disk_size | Size of the disk in GB | 30 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
 | controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
 | worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |

docs/fedora-coreos/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.20.5 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.21.2 cluster on bare-metal with Fedora CoreOS.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -111,7 +111,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -138,7 +138,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     matchbox = {
       source = "poseidon/matchbox"
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -283,9 +283,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.20.5
-node2.example.com   Ready   <none>  10m  v1.20.5
-node3.example.com   Ready   <none>  10m  v1.20.5
+node1.example.com   Ready   <none>  10m  v1.21.2
+node2.example.com   Ready   <none>  10m  v1.21.2
+node3.example.com   Ready   <none>  10m  v1.21.2
 ```
 
 List the pods.

docs/fedora-coreos/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on DigitalOcean with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -29,7 +29,7 @@ cd infra/clusters
 
 ## Provider
 
-Login to [DigitalOcean](https://cloud.digitalocean.com) or create an [account](https://cloud.digitalocean.com/registrations/new), if you don't have one.
+Login to [DigitalOcean](https://cloud.digitalocean.com). Or if you don't have one, create an account with our [referral link](https://m.do.co/c/94a5a4e76387) to get free credits.
 
 Generate a Personal Access Token with read/write scope from the [API tab](https://cloud.digitalocean.com/settings/api/tokens). Write the token to a file that can be referenced in configs.
 
@@ -51,7 +51,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     digitalocean = {
       source = "digitalocean/digitalocean"
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.20.5
-10.132.115.81      Ready   <none>  10m  v1.20.5
-10.132.124.107     Ready   <none>  10m  v1.20.5
+10.132.110.130     Ready   <none>  10m  v1.21.2
+10.132.115.81      Ready   <none>  10m  v1.21.2
+10.132.124.107     Ready   <none>  10m  v1.21.2
 ```
 
 List the pods.

docs/fedora-coreos/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on Google Compute Engine with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -52,11 +52,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     google = {
       source = "hashicorp/google"
-      version = "3.60.0"
+      version = "3.66.1"
     }
   }
 }
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.5
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.5
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.5
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.2
 ```
 
 List the pods.
@@ -216,7 +216,7 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "stable", "testing", "next" |
-| disk_size | Size of the disk in GB | 40 | 100 |
+| disk_size | Size of the disk in GB | 30 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
 | worker_snippets | Worker Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |

docs/flatcar-linux/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on AWS with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -51,11 +51,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     aws = {
       source = "hashicorp/aws"
-      version = "3.33.0"
+      version = "3.39.0"
     }
   }
 }
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.21.2"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.20.5
-ip-10-0-26-65  Ready   <none>  10m  v1.20.5
-ip-10-0-41-21  Ready   <none>  10m  v1.20.5
+ip-10-0-3-155  Ready   <none>  10m  v1.21.2
+ip-10-0-26-65  Ready   <none>  10m  v1.21.2
+ip-10-0-41-21  Ready   <none>  10m  v1.21.2
 ```
 
 List the pods.
@@ -211,7 +211,7 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
 | os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
-| disk_size | Size of the EBS volume in GB | 40 | 100 |
+| disk_size | Size of the EBS volume in GB | 30 | 100 |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |

docs/flatcar-linux/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on Azure with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -48,11 +48,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     azurerm = {
       source = "hashicorp/azurerm"
-      version = "2.52.0"
+      version = "2.58.0"
     }
   }
 }
@@ -75,7 +75,7 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.21.2"
 
   # Azure
   cluster_name   = "ramius"
@@ -149,9 +149,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.20.5
-ramius-worker-000001  Ready   <none>  25m  v1.20.5
-ramius-worker-000002  Ready   <none>  24m  v1.20.5
+ramius-controller-0   Ready   <none>  24m  v1.21.2
+ramius-worker-000001  Ready   <none>  25m  v1.21.2
+ramius-worker-000002  Ready   <none>  24m  v1.21.2
 ```
 
 List the pods.
@@ -229,7 +229,7 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | controller_type | Machine type for controllers | "Standard_B2s" | See below |
 | worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
 | os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
-| disk_size | Size of the disk in GB | 40 | 100 |
+| disk_size | Size of the disk in GB | 30 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |

docs/flatcar-linux/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.20.5 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.21.2 cluster on bare-metal with Flatcar Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -111,7 +111,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -138,7 +138,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     matchbox = {
       source = "poseidon/matchbox"
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.21.2"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -293,9 +293,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.20.5
-node2.example.com   Ready   <none>  10m  v1.20.5
-node3.example.com   Ready   <none>  10m  v1.20.5
+node1.example.com   Ready   <none>  10m  v1.21.2
+node2.example.com   Ready   <none>  10m  v1.21.2
+node3.example.com   Ready   <none>  10m  v1.21.2
 ```
 
 List the pods.

docs/flatcar-linux/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on DigitalOcean with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -29,7 +29,7 @@ cd infra/clusters
 
 ## Provider
 
-Login to [DigitalOcean](https://cloud.digitalocean.com) or create an [account](https://cloud.digitalocean.com/registrations/new), if you don't have one.
+Login to [DigitalOcean](https://cloud.digitalocean.com). Or if you don't have one, create an account with our [referral link](https://m.do.co/c/94a5a4e76387) to get free credits.
 
 Generate a Personal Access Token with read/write scope from the [API tab](https://cloud.digitalocean.com/settings/api/tokens). Write the token to a file that can be referenced in configs.
 
@@ -51,7 +51,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     digitalocean = {
       source = "digitalocean/digitalocean"
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.21.2"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.20.5
-10.132.115.81      Ready   <none>  10m  v1.20.5
-10.132.124.107     Ready   <none>  10m  v1.20.5
+10.132.110.130     Ready   <none>  10m  v1.21.2
+10.132.115.81      Ready   <none>  10m  v1.21.2
+10.132.124.107     Ready   <none>  10m  v1.21.2
 ```
 
 List the pods.

docs/flatcar-linux/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.20.5 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.21.2 cluster on Google Compute Engine with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -52,11 +52,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.8.0"
     }
     google = {
       source = "hashicorp/google"
-      version = "3.60.0"
+      version = "3.66.1"
     }
   }
 }
@@ -92,7 +92,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.21.2"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -167,9 +167,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.5
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.5
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.5
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.2
 ```
 
 List the pods.
@@ -236,7 +236,7 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
-| disk_size | Size of the disk in GB | 40 | 100 |
+| disk_size | Size of the disk in GB | 30 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@@ -29,7 +29,11 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](fedora-coreos/azure.md) | alpha |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | stable |
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](fedora-coreos/digitalocean.md) | beta |
-| Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](fedora-coreos/google-cloud/kubernetes) | stable |
+| Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](fedora-coreos/google-cloud.md) | stable |
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora CoreOS (ARM64) | [aws/fedora-coreos/kubernetes](advanced/arm64.md) | alpha |
 
 Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).
 
@@ -53,7 +57,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.2"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -91,9 +95,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.5
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.5
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.5
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.2
 ```
 
 List the pods.
@@ -116,7 +120,7 @@ kube-system   kube-scheduler-controller-0               1/1    Running   0
 
 ## Help
 
-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
+Schedule a meeting via [Github Sponsors](https://github.com/sponsors/poseidon?frequency=one-time) to discuss your use case.
 
 ## Motivation
 

docs/topics/maintenance.md

@@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar
 
 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.2"
   ...
 }
 
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.21.2"
   ...
 }
 ```
@@ -140,8 +140,8 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
--     version = "0.6.1"
-+     version = "0.7.1"
+-     version = "0.7.1"
++     version = "0.8.0"
     }
     ...
   }
@@ -193,12 +193,14 @@ Google Cloud creates a new worker template and edits the worker instance group i
 
 Terraform [v0.13](https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-13) introduced major changes to the provider plugin system. Terraform `init` can automatically install both `hashicorp` and `poseidon` provider plugins, eliminating the need to manually install plugin binaries.
 
-Typhoon modules have been updated for v0.13.x, but retain compatibility with v0.12.26+ to ease migration. Poseidon publishes [providers](/topics/security/#terraform-providers) to the Terraform Provider Registry for usage with v0.13+.
+Typhoon modules have been updated for v0.13.x. Poseidon publishes [providers](/topics/security/#terraform-providers) to the Terraform Provider Registry for usage with v0.13+.
 
 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.20.5 - ?       | v0.13.x, v0.14.4+   |
-| v1.20.0 - v1.20.5 | v0.13.x             |
+| v1.21.2 - ?       | v0.13.x, v0.14.4+, v0.15.x, v1.0.x |
+| v1.21.1 - v1.21.1 | v0.13.x, v0.14.4+, v0.15.x |
+| v1.20.2 - v1.21.0 | v0.13.x, v0.14.4+   |
+| v1.20.0 - v1.20.2 | v0.13.x             |
 | v1.18.8 - v1.19.4 | v0.12.26+, v0.13.x  |
 | v1.15.0 - v1.18.8 | v0.12.x             |
 | v1.10.3 - v1.15.0 | v0.11.x             |
@@ -208,7 +210,7 @@ Typhoon modules have been updated for v0.13.x, but retain compatibility with v0.
 
 ### New Workspace
 
-With a new Terraform workspace, use Terraform v0.13.x and the updated Typhoon [tutorials](/fedora-coreos/aws/#provider).
+With a new Terraform workspace, use Terraform v0.15.x and the updated Typhoon [tutorials](/fedora-coreos/aws/#provider).
 
 ### Existing Workspace
 

google-cloud/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

google-cloud/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -12,6 +12,7 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 
   trusted_certs_dir = "/etc/pki/tls/certs"
 

google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -51,8 +51,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -63,12 +63,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -86,7 +86,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -118,7 +117,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.21.2
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

google-cloud/fedora-coreos/kubernetes/variables.tf

@@ -60,7 +60,7 @@ variable "os_stream" {
 variable "disk_size" {
   type        = number
   description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "worker_preemptible" {
@@ -136,3 +136,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

google-cloud/fedora-coreos/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     google   = ">= 2.19, < 4.0"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.2.0
 systemd:
   units:
     - name: docker.service
@@ -24,8 +24,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -36,12 +36,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -59,7 +59,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -67,6 +66,9 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
           --rotate-certificates \
@@ -83,7 +85,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

google-cloud/fedora-coreos/kubernetes/workers/variables.tf

@@ -48,7 +48,7 @@ variable "os_stream" {
 variable "disk_size" {
   type        = number
   description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "preemptible" {
@@ -90,6 +90,12 @@ variable "node_labels" {
   default     = []
 }
 
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported, temporary
 
 variable "cluster_domain_suffix" {

google-cloud/fedora-coreos/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     google   = ">= 2.19, < 4.0"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.8"
     }
   }
 }

google-cloud/fedora-coreos/kubernetes/workers/workers.tf

@@ -89,6 +89,7 @@ data "template_file" "worker-config" {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

google-cloud/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.21.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

google-cloud/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=c0718e85529bbdd53bb83b2ce41904710c6bf6f5"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -12,6 +12,7 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 
   // temporary
   external_apiserver_port = 443

google-cloud/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -55,8 +55,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -67,6 +67,7 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
@@ -87,7 +88,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -115,7 +115,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.2
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \