Update nginx-ingess and Prometheus exporter addons

* Update flannel from v0.15.0 to v0.15.1
* https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: [poseidon]

.github/dependabot.yaml

@@ -0,0 +1,9 @@
+version: 2
+updates:
+- package-ecosystem: pip
+  directory: "/"
+  schedule:
+    interval: weekly
+  pull-request-branch-name:
+    separator: "-"
+  open-pull-requests-limit: 3

CHANGES.md

@@ -4,6 +4,261 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.22.4
+
+* Kubernetes [v1.22.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224)
+* Update CoreDNS from v1.8.4 to [v1.8.6](https://github.com/poseidon/terraform-render-bootstrap/pull/284)
+* Update Calico from v3.20.2 to [v3.21.0](https://github.com/projectcalico/calico/releases/tag/v3.21.0)
+* Update flannel from v0.14.0 to [v0.15.1](https://github.com/flannel-io/flannel/releases/tag/v0.15.1)
+
+### Google
+
+* Allow use of Terraform provider `google` [v4.0+](https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0)
+
+### Flatcar Linux
+
+* Change Kubelet mounts for cgroups v2 ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+* Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+
+### Addons
+
+* Update Prometheus from v2.30.3 to [v2.31.1](https://github.com/prometheus/prometheus/releases/tag/v2.31.1)
+* Update node-exporter from v1.2.2 to [v1.3.0](https://github.com/prometheus/node_exporter/releases/tag/v1.3.0)
+* Update kube-state-metrics from v2.2.3 to [v2.2.4](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.4)
+* Update Grafana from v8.2.1 to [v8.2.4](https://github.com/grafana/grafana/releases/tag/v8.2.4)
+* Update nginx-ingress from v1.0.4 to [v1.0.5](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.5)
+
+## v1.23.3
+
+* Kubernetes [v1.22.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223)
+* Update etcd from v3.5.0 to [v3.5.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.1)
+* Update Cilium from v1.10.4 to [v1.10.5](https://github.com/cilium/cilium/releases/tag/v1.10.5)
+* Update Calico from v3.20.1 to [v3.20.2](https://github.com/projectcalico/calico/releases/tag/v3.20.2)
+  * Use Calico's iptables legacy vs nft auto-detection
+* Update flannel from v0.13.0 to v0.14.0
+* Change `enable_aggregation` default to true ([#279](https://github.com/poseidon/terraform-render-bootstrap/pull/279))
+
+### Bare-Metal
+
+* Require Terraform provider `poseidon/matchbox` v0.5+ ([#1048](https://github.com/poseidon/typhoon/pull/1048))
+
+### Addons
+
+* Update nginx-ingress from v1.0.0 to [v1.0.4](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4)
+* Update Prometheus from v2.29.2 to [v2.30.3](https://github.com/prometheus/prometheus/releases/tag/v2.30.3)
+* Update kube-state-metrics from v2.2.0 to [v2.2.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.3)
+* Update Grafana from v8.1.2 to [v8.2.1](https://github.com/grafana/grafana/releases/tag/v8.2.1)
+
+## v1.22.2
+
+* Kubernetes [v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222)
+* Update Cilium from v1.10.3 to [v1.10.4](https://github.com/cilium/cilium/releases/tag/v1.10.4)
+* Update Calico from v3.20.0 to [v3.20.1](https://github.com/projectcalico/calico/releases/tag/v3.20.1)
+* Fix access to ClusterIP services with Cilium ([#276](https://github.com/poseidon/terraform-render-bootstrap/pull/276))
+
+### Fedora CoreOS
+
+* Use Fedora CoreOS ARM64 AMIs ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+
+### Addons
+
+* Update Prometheus from v2.29.1 to [v2.29.2](https://github.com/prometheus/prometheus/releases/tag/v2.29.2)
+* Update kube-state-metrics from v2.1.1 to [v2.2.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.0)
+
+## v1.22.1
+
+* Kubernetes [v1.22.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221)
+* Update Calico from v3.19.1 to [v3.20.0](https://github.com/projectcalico/calico/releases/tag/v3.20.0)
+
+### Addons
+
+* Update nginx-ingress from v1.0.0-beta.1 to [v1.0.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0)
+* Update Prometheus from v2.28.1 to [v2.29.1](https://github.com/prometheus/prometheus/releases/tag/v2.29.1)
+* Update Grafana from v8.1.1 to [v8.1.2](https://github.com/grafana/grafana/releases/tag/v8.1.2)
+
+## v1.22.0
+
+* Kubernetes [v1.22.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1220)
+* Update etcd from v3.4.16 to [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0)
+* Switch `kube-controller-manager` and `kube-scheduler` to use secure port only
+  * Update Prometheus config to discover endpoints and use a bearer token to scrape
+
+### Fedora CoreOS
+
+* Add Cilium cgroups v2 support on Fedora CoreOS
+* Update Butane Config version from v1.2.0 to v1.4.0
+  * Rename Fedora CoreOS Config to Butane Config
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.4.0
+
+### Addons
+
+* Update nginx-ingress from v0.47.0 to [v1.0.0-beta.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0-beta.1)
+* Update node-exporter from v1.2.0 to [v1.2.2](https://github.com/prometheus/node_exporter/releases/tag/v1.2.2)
+* Update kube-state-metrics from v2.1.0 to [v2.1.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.1)
+* Update Grafana from v8.0.6 to [v8.1.1](https://github.com/grafana/grafana/releases/tag/v8.1.1)
+
+## v1.21.3
+
+* Kubernetes [v1.21.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1213)
+* Update Cilium from v1.10.1 to [v1.10.3](https://github.com/cilium/cilium/releases/tag/v1.10.3)
+* Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.9+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+
+### AWS
+
+* Change default disk type from `gp2` to `gp3` ([#1012](https://github.com/poseidon/typhoon/pull/1012))
+
+### Addons
+
+* Update Prometheus from v2.28.0 to [v2.28.1](https://github.com/prometheus/prometheus/releases/tag/v2.28.1)
+* Update node-exporter from v1.1.2 to [v1.2.0](https://github.com/prometheus/node_exporter/releases/tag/v1.2.0)
+* Update Grafana from v8.0.3 to [v8.0.6](https://github.com/grafana/grafana/releases/tag/v8.0.6)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
+## v1.21.2
+
+* Kubernetes [v1.21.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1212)
+* Add Terraform v1.0.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x, v0.14.4+, and v0.15.x
+* Update CoreDNS from v1.8.0 to [v1.8.4]([#1006](https://github.com/poseidon/typhoon/pull/1006))
+* Update Cilium from v1.9.6 to [v1.10.1](https://github.com/cilium/cilium/releases/tag/v1.10.1)
+* Update Calico from v3.19.0 to [v3.19.1](https://github.com/projectcalico/calico/releases/tag/v3.19.1)
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0 to [v2.1.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.0)
+* Update Prometheus from v2.27.0 to [v2.28.0](https://github.com/prometheus/prometheus/releases/tag/v2.28.0)
+* Update Grafana from v7.5.6 to [v8.0.3](https://github.com/grafana/grafana/releases/tag/v8.0.3)
+* Update nginx-ingress from v0.46.0 to [v0.47.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.47.0)
+
+### Fedora CoreOS
+
+#### AWS
+
+* Extend experimental Fedora CoreOS arm64 support with Cilium
+  * CNI provider may now be `flannel` or `cilium` (new)
+
+#### Bare-Metal
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+#### DigitalOcean
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
+## v1.21.1
+
+* Kubernetes [v1.21.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211)
+* Add Terraform v0.15.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x and v0.14.4+
+* Update etcd from v3.4.15 to [v3.4.16](https://github.com/etcd-io/etcd/releases/tag/v3.4.16)
+* Update Cilium from v1.9.5 to [v1.9.6](https://github.com/cilium/cilium/releases/tag/v1.9.6)
+* Update Calico from v3.18.1 to [v3.19.0](https://github.com/projectcalico/calico/releases/tag/v3.19.0)
+
+### AWS
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Azure
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Google Cloud
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Fedora CoreOS
+
+* Update Kubelet mounts for cgroups v2 ([#978](https://github.com/poseidon/typhoon/pull/978))
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0-rc.1 to [v2.0.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0)
+* Update Prometheus from v2.25.2 to [v2.27.0](https://github.com/prometheus/prometheus/releases/tag/v2.27.0)
+* Update Grafana from v7.5.3 to [v7.5.6](https://github.com/grafana/grafana/releases/tag/v7.5.6)
+* Update nginx-ingress from v0.45.0 to [v0.46.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.46.0)
+
+## v1.21.0
+
+* Kubernetes [v1.21.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1210)
+  * Enable `tokencleaner` controller ([#969](https://github.com/poseidon/typhoon/pull/969))
+  * Enable `kube-scheduler` and `kube-controller-manager` separate authn/z kubeconfig
+  * Change CNI config location from /etc/kubernetes/cni/net.d to /etc/cni/net.d ([#965](https://github.com/poseidon/typhoon/pull/965))
+  * Change `kube-controller-manager` to mount `/var/lib/kubelet/volumeplugins` directly
+  * Remove unused `cloud-provider` flags
+* Update Fedora CoreOS Config version from v1.1.0 to v1.2.0 ([#970](https://github.com/poseidon/typhoon/pull/970))
+  * Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.8+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.2.0
+
+### AWS
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Azure
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+* Remove deprecated `azurerm_lb_backend_address_pool` field `resource_group_name` ([#972](https://github.com/poseidon/typhoon/pull/972))
+
+### Google Cloud
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Addons
+
+* Update nginx-ingress from v0.44.0 to [v0.45.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.45.0)
+* Update kube-state-metrics from v2.0.0-rc.0 to [v2.0.0-rc.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-rc.1)
+* Update Grafana from v7.4.5 to [v7.5.3](https://github.com/grafana/grafana/releases/tag/v7.5.3)
+
+## v1.20.5
+
+* Kubernetes [v1.20.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1205)
+* Update etcd from v3.4.14 to [v3.4.15](https://github.com/etcd-io/etcd/releases/tag/v3.4.15)
+* Update Cilium from v1.9.4 to [v1.9.5](https://github.com/cilium/cilium/releases/tag/v1.9.5)
+* Update Calico from v3.17.3 to [v3.18.1](https://github.com/projectcalico/calico/releases/tag/v3.18.1)
+* Update CoreDNS from v1.7.0 to [v1.8.0](https://coredns.io/2020/10/22/coredns-1.8.0-release/)
+* Mark bootstrap token as sensitive in Terraform plans ([#949](https://github.com/poseidon/typhoon/pull/949))
+
+### Fedora CoreOS
+
+* Set Kubelet `provider-id` ([#951](https://github.com/poseidon/typhoon/pull/951))
+
+### Flatcar Linux
+
+#### AWS
+
+* Set Kubelet `provider-id` ([#951](https://github.com/poseidon/typhoon/pull/951))
+* Remove `os_image` option `flatcar-edge` ([#943](https://github.com/poseidon/typhoon/pull/943))
+
+#### Azure
+
+* Remove `os_image` option `flatcar-edge` ([#943](https://github.com/poseidon/typhoon/pull/943))
+
+#### Bare-Metal
+
+* Remove `os_channel` option `flatcar-edge` ([#943](https://github.com/poseidon/typhoon/pull/943))
+
+### Addons
+
+* Update Prometheus from v2.25.0 to [v2.25.2](https://github.com/prometheus/prometheus/releases/tag/v2.25.2)
+* Update kube-state-metrics from v2.0.0-alpha.3 to [v2.0.0-rc.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-rc.0)
+  * Switch image from `quay.io` to `k8s.gcr.io` ([#946](https://github.com/poseidon/typhoon/pull/946))
+* Update node-exporter from v1.1.1 to [v1.1.2](https://github.com/prometheus/node_exporter/releases/tag/v1.1.2)
+* Update Grafana from v7.4.2 to [v7.4.5](https://github.com/grafana/grafana/releases/tag/v7.4.5)
+
+## v1.20.4
+
 * Kubernetes [v1.20.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1204)
 * Update Cilium from v1.9.1 to [v1.9.4](https://github.com/cilium/cilium/releases/tag/v1.9.4)
 * Update Calico from v3.17.1 to [v3.17.3](https://github.com/projectcalico/calico/releases/tag/v3.17.3)

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@@ -31,6 +31,10 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | stable |
 
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora CoreOS (ARM64) | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | alpha |
+
 Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).
 
 | Platform      | Operating System | Terraform Module | Status |
@@ -54,7 +58,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.20.4"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.4"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -63,7 +67,7 @@ module "yavin" {
   dns_zone_name = "example-zone"
 
   # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
   # optional
   worker_count = 2
@@ -93,9 +97,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.4
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.4
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.4
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.4
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.4
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.4
 ```
 
 List the pods.
@@ -126,7 +130,7 @@ Typhoon is strict about minimalism, maturity, and scope. These are not in scope:
 
 ## Help
 
-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
+Schedule a meeting via [Github Sponsors](https://github.com/sponsors/poseidon?frequency=one-time) to discuss your use case.
 
 ## Motivation
 
@@ -142,6 +146,11 @@ Typhoon clusters will contain only [free](https://www.debian.org/intro/free) com
 
 ## Sponsors
 
-Poseidon's Github [Sponsors](https://github.com/sponsors/poseidon) support the infrastructure and operational costs of providing Typhoon. If you'd like your company here, please contact dghubble at psdn.io.
+Poseidon's Github [Sponsors](https://github.com/sponsors/poseidon) support the infrastructure and operational costs of providing Typhoon.
 
-* [DigitalOcean](https://www.digitalocean.com/) kindly provides us cloud credits
+<a href="https://www.digitalocean.com/">
+    <img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_blue.svg" width="201px">
+</a>
+<br>
+
+If you'd like your company here, please contact dghubble at psdn.io.

addons/grafana/deployment.yaml

@@ -24,7 +24,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: grafana
-          image: docker.io/grafana/grafana:7.4.2
+          image: docker.io/grafana/grafana:8.2.4
           env:
             - name: GF_PATHS_CONFIG
               value: "/etc/grafana/custom.ini"

addons/nginx-ingress/aws/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.0.5
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/azure/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.0.5
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.0.5
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.0.5
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.0.5
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/prometheus/config.yaml

@@ -72,6 +72,48 @@ data:
         regex: apiserver_request_duration_seconds_count;.+
         action: drop
 
+    # Scrape config for kube-controller-manager endpoints.
+    #
+    # kube-controller-manager service endpoints can be discovered by using the
+    # `endpoints` role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-controller-manager and the `https` port.
+    - job_name: 'kube-controller-manager'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-controller-manager;metrics
+      - replacement: kube-controller-manager
+        action: replace
+        target_label: job
+
+    # Scrape config for kube-scheduler endpoints.
+    #
+    # kube-scheduler service endpoints can be discovered by using the `endpoints`
+    # role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-scheduler and the `https` port.
+    - job_name: 'kube-scheduler'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-scheduler;metrics
+      - replacement: kube-scheduler
+        action: replace
+        target_label: job
+
     # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
     # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
     - job_name: 'kubelet'

addons/prometheus/deployment.yaml

@@ -21,7 +21,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.25.0
+          image: quay.io/prometheus/prometheus:v2.31.1
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/discovery/kube-controller-manager.yaml

@@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
   name: kube-controller-manager
   namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
   type: ClusterIP
   clusterIP: None
@@ -14,5 +12,5 @@ spec:
   ports:
     - name: metrics
       protocol: TCP
-      port: 10252
-      targetPort: 10252
+      port: 10257
+      targetPort: 10257

addons/prometheus/discovery/kube-scheduler.yaml

@@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
   name: kube-scheduler
   namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
   type: ClusterIP
   clusterIP: None
@@ -14,5 +12,5 @@ spec:
   ports:
     - name: metrics
       protocol: TCP
-      port: 10251
-      targetPort: 10251
+      port: 10259
+      targetPort: 10259

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -25,7 +25,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v2.0.0-alpha.3
+        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.2.4
         ports:
           - name: metrics
             containerPort: 8080

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -28,13 +28,13 @@ spec:
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.1.1
+        image: quay.io/prometheus/node-exporter:v1.3.0
         args:
           - --path.procfs=/host/proc
           - --path.sysfs=/host/sys
           - --path.rootfs=/host/root
-          - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
-          - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
+          - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)($|/)
+          - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
         ports:
           - name: metrics
             containerPort: 9100

aws/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/fedora-coreos/kubernetes/ami.tf

@@ -18,15 +18,11 @@ data "aws_ami" "fedora-coreos" {
     values = ["Fedora CoreOS ${var.os_stream} *"]
   }
 }
-
-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
   count = var.arch == "arm64" ? 1 : 0
 
   most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]
 
   filter {
     name   = "architecture"
@@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
   }
 
   filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
   }
 }
-

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/fedora-coreos/kubernetes/controllers.tf

@@ -62,7 +62,6 @@ data "template_file" "controller-configs" {
 
   vars = {
     # Cannot use cyclic dependencies on controllers or their DNS records
-    etcd_arch   = var.arch == "arm64" ? "-arm64" : ""
     etcd_name   = "etcd${count.index}"
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...

aws/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14${etcd_arch}
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -50,10 +50,13 @@ systemd:
       contents: |
         [Unit]
         Description=Kubelet (System Container)
+        Requires=afterburn.service
+        After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        EnvironmentFile=/run/metadata/afterburn
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -64,12 +67,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -87,12 +90,12 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
           --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
@@ -119,7 +122,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.4
+            quay.io/poseidon/kubelet:v1.22.4
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -215,6 +218,7 @@ storage:
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
           ETCD_UNSUPPORTED_ARCH=arm64
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

aws/fedora-coreos/kubernetes/security.tf

@@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {
 
   type                     = "ingress"
   protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
   source_security_group_id = aws_security_group.worker.id
 }
 
@@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {
 
   type                     = "ingress"
   protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
   source_security_group_id = aws_security_group.worker.id
 }
 

aws/fedora-coreos/kubernetes/variables.tf

@@ -55,13 +55,13 @@ variable "os_stream" {
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
   type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }
 
 variable "disk_iops" {
@@ -84,13 +84,13 @@ variable "worker_target_groups" {
 
 variable "controller_snippets" {
   type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
   default     = []
 }
 
 variable "worker_snippets" {
   type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
   default     = []
 }
 
@@ -176,4 +176,3 @@ variable "daemonset_tolerations" {
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
   default     = []
 }
-

aws/fedora-coreos/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: docker.service
@@ -23,10 +23,13 @@ systemd:
       contents: |
         [Unit]
         Description=Kubelet (System Container)
+        Requires=afterburn.service
+        After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        EnvironmentFile=/run/metadata/afterburn
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -37,12 +40,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -60,7 +63,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -72,6 +74,7 @@ systemd:
           --register-with-taints=${taint} \
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
           --read-only-port=0 \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -87,7 +90,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -126,6 +129,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

aws/fedora-coreos/kubernetes/workers/variables.tf

@@ -48,13 +48,13 @@ variable "os_stream" {
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
   type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }
 
 variable "disk_iops" {
@@ -77,7 +77,7 @@ variable "target_groups" {
 
 variable "snippets" {
   type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
   default     = []
 }
 

aws/fedora-coreos/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

aws/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -12,5 +12,6 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }
 

aws/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -53,11 +53,13 @@ systemd:
         Description=Kubelet (System Container)
         Requires=docker.service
         After=docker.service
+        Requires=coreos-metadata.service
+        After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        EnvironmentFile=/run/metadata/coreos
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -68,13 +70,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -85,16 +87,16 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
           --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
@@ -117,7 +119,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

aws/flatcar-linux/kubernetes/controllers.tf

@@ -67,7 +67,6 @@ data "template_file" "controller-configs" {
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
-    cgroup_driver          = local.channel == "edge" ? "systemd" : "cgroupfs"
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)

aws/flatcar-linux/kubernetes/security.tf

@@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {
 
   type                     = "ingress"
   protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
   source_security_group_id = aws_security_group.worker.id
 }
 
@@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {
 
   type                     = "ingress"
   protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
   source_security_group_id = aws_security_group.worker.id
 }
 

aws/flatcar-linux/kubernetes/variables.tf

@@ -43,25 +43,25 @@ variable "worker_type" {
 
 variable "os_image" {
   type        = string
-  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
   default     = "flatcar-stable"
 
   validation {
-    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
-    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, or flatcar-alpha."
   }
 }
 
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
   type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }
 
 variable "disk_iops" {
@@ -160,3 +160,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

aws/flatcar-linux/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

aws/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -25,11 +25,13 @@ systemd:
         Description=Kubelet
         Requires=docker.service
         After=docker.service
+        Requires=coreos-metadata.service
+        After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        EnvironmentFile=/run/metadata/coreos
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -43,13 +45,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -60,11 +62,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -72,7 +73,11 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
           --read-only-port=0 \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -89,7 +94,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

aws/flatcar-linux/kubernetes/workers/variables.tf

@@ -36,25 +36,25 @@ variable "instance_type" {
 
 variable "os_image" {
   type        = string
-  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
   default     = "flatcar-stable"
 
   validation {
-    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
-    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, or flatcar-alpha."
   }
 }
 
 variable "disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "disk_type" {
   type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }
 
 variable "disk_iops" {
@@ -113,3 +113,9 @@ variable "node_labels" {
   description = "List of initial node labels"
   default     = []
 }
+
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}

aws/flatcar-linux/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

aws/flatcar-linux/kubernetes/workers/workers.tf

@@ -85,8 +85,8 @@ data "template_file" "worker-config" {
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
-    cgroup_driver          = local.channel == "edge" ? "systemd" : "cgroupfs"
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

azure/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -18,6 +18,7 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 
   # Fedora CoreOS
   trusted_certs_dir = "/etc/pki/tls/certs"

azure/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -51,8 +51,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -63,12 +63,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -86,7 +86,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -118,7 +117,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.4
+            quay.io/poseidon/kubelet:v1.22.4
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -213,6 +212,7 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

azure/fedora-coreos/kubernetes/lb.tf

@@ -112,16 +112,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {
 
 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "controller"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "worker"
   loadbalancer_id = azurerm_lb.cluster.id
 }

azure/fedora-coreos/kubernetes/security.tf

@@ -95,7 +95,7 @@ resource "azurerm_network_security_rule" "controller-kube-metrics" {
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
-  destination_port_range      = "10251-10252"
+  destination_port_range      = "10257-10259"
   source_address_prefix       = azurerm_subnet.worker.address_prefix
   destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

azure/fedora-coreos/kubernetes/variables.tf

@@ -54,7 +54,7 @@ variable "os_image" {
 variable "disk_size" {
   type        = number
   description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "worker_priority" {
@@ -65,13 +65,13 @@ variable "worker_priority" {
 
 variable "controller_snippets" {
   type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
   default     = []
 }
 
 variable "worker_snippets" {
   type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
   default     = []
 }
 
@@ -135,3 +135,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

azure/fedora-coreos/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: docker.service
@@ -24,8 +24,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -36,12 +36,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -59,7 +59,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -67,6 +66,9 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
           --rotate-certificates \
@@ -83,7 +85,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -122,6 +124,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

azure/fedora-coreos/kubernetes/workers/variables.tf

@@ -57,7 +57,7 @@ variable "priority" {
 
 variable "snippets" {
   type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
   default     = []
 }
 
@@ -88,6 +88,12 @@ variable "node_labels" {
   default     = []
 }
 
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported
 
 variable "cluster_domain_suffix" {

azure/fedora-coreos/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

azure/fedora-coreos/kubernetes/workers/workers.tf

@@ -87,6 +87,7 @@ data "template_file" "worker-config" {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

azure/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -18,5 +18,6 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }
 

azure/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -55,9 +55,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -68,13 +67,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -85,11 +84,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -117,7 +115,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

azure/flatcar-linux/kubernetes/controllers.tf

@@ -150,7 +150,6 @@ data "template_file" "controller-configs" {
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
-    cgroup_driver          = local.channel == "edge" ? "systemd" : "cgroupfs"
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)

azure/flatcar-linux/kubernetes/lb.tf

@@ -112,16 +112,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {
 
 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "controller"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name            = "worker"
   loadbalancer_id = azurerm_lb.cluster.id
 }

azure/flatcar-linux/kubernetes/security.tf

@@ -95,7 +95,7 @@ resource "azurerm_network_security_rule" "controller-kube-metrics" {
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
-  destination_port_range      = "10251-10252"
+  destination_port_range      = "10257-10259"
   source_address_prefix       = azurerm_subnet.worker.address_prefix
   destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

azure/flatcar-linux/kubernetes/variables.tf

@@ -48,19 +48,19 @@ variable "worker_type" {
 
 variable "os_image" {
   type        = string
-  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
   default     = "flatcar-stable"
 
   validation {
-    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
-    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, or flatcar-alpha."
   }
 }
 
 variable "disk_size" {
   type        = number
   description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }
 
 variable "worker_priority" {
@@ -141,3 +141,8 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}

azure/flatcar-linux/kubernetes/versions.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
@@ -9,7 +9,7 @@ terraform {
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

azure/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -27,9 +27,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -43,13 +42,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -60,11 +59,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
           --network-plugin=cni \
@@ -72,6 +70,9 @@ systemd:
           %{~ for label in split(",", node_labels) ~}
           --node-labels=${label} \
           %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
           --rotate-certificates \
@@ -89,7 +90,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

azure/flatcar-linux/kubernetes/workers/variables.tf

@@ -46,12 +46,12 @@ variable "vm_type" {
 
 variable "os_image" {
   type        = string
-  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
   default     = "flatcar-stable"
 
   validation {
-    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_image)
-    error_message = "The os_image must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, or flatcar-alpha."
   }
 }
 
@@ -94,6 +94,12 @@ variable "node_labels" {
   default     = []
 }
 
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported
 
 variable "cluster_domain_suffix" {

azure/flatcar-linux/kubernetes/workers/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
     template = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
   }
 }

azure/flatcar-linux/kubernetes/workers/workers.tf

@@ -104,8 +104,8 @@ data "template_file" "worker-config" {
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
     cluster_domain_suffix  = var.cluster_domain_suffix
-    cgroup_driver          = local.channel == "edge" ? "systemd" : "cgroupfs"
     node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
   }
 }
 

bare-metal/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -50,8 +50,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -62,12 +62,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -85,7 +85,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -120,6 +119,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
@@ -127,7 +127,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.4
+            $${KUBELET_IMAGE}
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -222,6 +222,7 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: docker.service
@@ -23,8 +23,8 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -35,12 +35,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -58,7 +58,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -121,6 +120,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

bare-metal/fedora-coreos/kubernetes/ssh.tf

@@ -39,6 +39,7 @@ resource "null_resource" "copy-controller-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
       "sudo /opt/bootstrap/layout",
     ]
   }
@@ -70,6 +71,7 @@ resource "null_resource" "copy-worker-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
     ]
   }
 }

bare-metal/fedora-coreos/kubernetes/variables.tf

@@ -57,7 +57,7 @@ EOD
 
 variable "snippets" {
   type        = map(list(string))
-  description = "Map from machine names to lists of Fedora CoreOS Config snippets"
+  description = "Map from machine names to lists of Butane snippets"
   default     = {}
 }
 

bare-metal/fedora-coreos/kubernetes/versions.tf

@@ -1,19 +1,19 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
 
     matchbox = {
       source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
     }
   }
 }

bare-metal/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -63,9 +63,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -76,13 +75,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -93,11 +92,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -126,7 +124,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

bare-metal/flatcar-linux/kubernetes/cl/worker.yaml

@@ -35,9 +35,8 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -51,13 +50,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -68,11 +67,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=${domain_name} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \

bare-metal/flatcar-linux/kubernetes/profiles.tf

@@ -106,7 +106,6 @@ data "template_file" "controller-configs" {
     domain_name            = var.controllers.*.domain[count.index]
     etcd_name              = var.controllers.*.name[count.index]
     etcd_initial_cluster   = join(",", formatlist("%s=https://%s:2380", var.controllers.*.name, var.controllers.*.domain))
-    cgroup_driver          = var.os_channel == "flatcar-edge" ? "systemd" : "cgroupfs"
     cluster_dns_service_ip = module.bootstrap.cluster_dns_service_ip
     cluster_domain_suffix  = var.cluster_domain_suffix
     ssh_authorized_key     = var.ssh_authorized_key
@@ -134,7 +133,6 @@ data "template_file" "worker-configs" {
 
   vars = {
     domain_name            = var.workers.*.domain[count.index]
-    cgroup_driver          = var.os_channel == "flatcar-edge" ? "systemd" : "cgroupfs"
     cluster_dns_service_ip = module.bootstrap.cluster_dns_service_ip
     cluster_domain_suffix  = var.cluster_domain_suffix
     ssh_authorized_key     = var.ssh_authorized_key

bare-metal/flatcar-linux/kubernetes/variables.tf

@@ -12,11 +12,11 @@ variable "matchbox_http_endpoint" {
 
 variable "os_channel" {
   type        = string
-  description = "Channel for a Flatcar Linux (flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "Channel for a Flatcar Linux (flatcar-stable, flatcar-beta, flatcar-alpha)"
 
   validation {
-    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"], var.os_channel)
-    error_message = "The os_channel must be flatcar-stable, flatcar-beta, flatcar-alpha, or flatcar-edge."
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_channel)
+    error_message = "The os_channel must be flatcar-stable, flatcar-beta, or flatcar-alpha."
   }
 }
 

bare-metal/flatcar-linux/kubernetes/versions.tf

@@ -1,19 +1,19 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
 
     matchbox = {
       source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
     }
   }
 }

digital-ocean/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: etcd-member.service
@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -52,9 +52,9 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -65,12 +65,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -88,7 +88,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -130,7 +129,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.4
+            quay.io/poseidon/kubelet:v1.22.4
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -220,3 +219,5 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+

digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml

@@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
   units:
     - name: docker.service
@@ -26,9 +26,9 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -39,12 +39,12 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
           --volume /etc/kubernetes:/etc/kubernetes:ro,z \
           --volume /usr/lib/os-release:/etc/os-release:ro \
           --volume /lib/modules:/lib/modules:ro \
           --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
           --volume /var/lib/calico:/var/lib/calico:ro \
           --volume /var/lib/docker:/var/lib/docker \
           --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
@@ -62,7 +62,6 @@ systemd:
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -93,7 +92,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -127,3 +126,4 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp

digital-ocean/fedora-coreos/kubernetes/network.tf

@@ -116,7 +116,7 @@ resource "digitalocean_firewall" "controllers" {
   # kube-scheduler metrics, kube-controller-manager metrics
   inbound_rule {
     protocol    = "tcp"
-    port_range  = "10251-10252"
+    port_range  = "10257-10259"
     source_tags = [digitalocean_tag.workers.name]
   }
 }

digital-ocean/fedora-coreos/kubernetes/ssh.tf

@@ -36,6 +36,7 @@ resource "null_resource" "copy-controller-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
       "sudo /opt/bootstrap/layout",
     ]
   }
@@ -60,6 +61,7 @@ resource "null_resource" "copy-worker-secrets" {
   provisioner "remote-exec" {
     inline = [
       "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
     ]
   }
 }
@@ -84,4 +86,3 @@ resource "null_resource" "bootstrap" {
     ]
   }
 }
-

digital-ocean/fedora-coreos/kubernetes/variables.tf

@@ -48,13 +48,13 @@ variable "os_image" {
 
 variable "controller_snippets" {
   type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
   default     = []
 }
 
 variable "worker_snippets" {
   type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
   default     = []
 }
 

digital-ocean/fedora-coreos/kubernetes/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
 
     digitalocean = {

digital-ocean/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.20.4 (upstream)
+* Kubernetes v1.22.4 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5633f97f75d2f2d05fe4f873e44733150ebc80d8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5353769db663814d3d91346068f3222709e1c0cd"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.14
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -65,9 +65,9 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -78,13 +78,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -95,10 +95,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -127,7 +127,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml

@@ -37,9 +37,9 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /var/lib/calico
@@ -53,13 +53,13 @@ systemd:
           --privileged \
           --pid host \
           --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
           -v /etc/kubernetes:/etc/kubernetes:ro \
           -v /etc/machine-id:/etc/machine-id:ro \
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -70,10 +70,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --healthz-port=0 \
           --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/var/lib/kubelet/kubeconfig \
@@ -96,7 +96,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.4
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.4
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

digital-ocean/flatcar-linux/kubernetes/network.tf

@@ -116,7 +116,7 @@ resource "digitalocean_firewall" "controllers" {
   # kube-scheduler metrics, kube-controller-manager metrics
   inbound_rule {
     protocol    = "tcp"
-    port_range  = "10251-10252"
+    port_range  = "10257-10259"
     source_tags = [digitalocean_tag.workers.name]
   }
 }

digital-ocean/flatcar-linux/kubernetes/versions.tf

@@ -1,14 +1,14 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     template = "~> 2.1"
     null     = "~> 2.1"
 
     ct = {
       source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
     }
 
     digitalocean = {

docs/advanced/arm64.md

@@ -6,14 +6,7 @@
 Typhoon has experimental support for ARM64 with Fedora CoreOS on AWS. Full clusters can be created with ARM64 controller and worker nodes. Or worker pools of ARM64 nodes can be attached to an AMD64 cluster to create a hybrid/mixed architecture cluster.
 
 !!! note
-    Currently, CNI networking must be set to flannel.
-
-## AMIs
-
-In lieu of official Fedora CoreOS ARM64 AMIs, Poseidon publishes experimental ARM64 AMIs to a few regions (us-east-1, us-east-2, us-west-1). These AMIs may be **removed** at any time and will be replaced when Fedora CoreOS publishes equivalents.
-
-!!! note
-    AMIs are only published to a few regions, and AWS availability of ARM instance types varies.
+    Currently, CNI networking must be set to flannel or Cilium.
 
 ## Cluster
 
@@ -21,7 +14,7 @@ Create a cluster with ARM64 controller and worker nodes. Container workloads mus
 
 ```tf
 module "gravitas" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.19.4"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.4"
 
   # AWS
   cluster_name = "gravitas"
@@ -29,11 +22,11 @@ module "gravitas" {
   dns_zone_id  = "Z3PAABBCFAKEC0"
 
   # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
   # optional
   arch         = "arm64"
-  networking   = "flannel"
+  networking   = "cilium"
   worker_count = 2
   worker_price = "0.0168"
 
@@ -47,9 +40,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes.
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-12-178   Ready    <none>   101s   v1.19.4   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-18-93    Ready    <none>   102s   v1.19.4   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-90-10    Ready    <none>   104s   v1.19.4   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-12-178   Ready    <none>   101s   v1.22.4   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-18-93    Ready    <none>   102s   v1.22.4   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-90-10    Ready    <none>   104s   v1.22.4   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
 ```
 
 ## Hybrid
@@ -60,7 +53,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.19.4"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.4"
 
       # AWS
       cluster_name = "gravitas"
@@ -68,10 +61,10 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
       dns_zone_id  = "Z3PAABBCFAKEC0"
 
       # configuration
-      ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
       # optional
-      networking   = "flannel"
+      networking   = "cilium"
       worker_count = 2
       worker_price = "0.021"
 
@@ -83,7 +76,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.19.4"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.22.4"
 
       # AWS
       vpc_id          = module.gravitas.vpc_id
@@ -107,10 +100,10 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 
 ```
 $ kubectl get nodes -o wide
-NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-14-73    Ready    <none>   116s   v1.19.4   10.0.14.73    <none>        Fedora CoreOS 32.20201018.3.0     5.8.15-201.fc32.x86_64    docker://19.3.11
-ip-10-0-17-167   Ready    <none>   104s   v1.19.4   10.0.17.167   <none>        Fedora CoreOS 32.20201018.3.0     5.8.15-201.fc32.x86_64    docker://19.3.11
-ip-10-0-47-166   Ready    <none>   110s   v1.19.4   10.0.47.166   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-7-237    Ready    <none>   111s   v1.19.4   10.0.7.237    <none>        Fedora CoreOS 32.20201018.3.0     5.8.15-201.fc32.x86_64    docker://19.3.11
+NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION             CONTAINER-RUNTIME
+ip-10-0-1-81    Ready    <none>   4m28s   v1.22.4   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-17-86   Ready    <none>   4m28s   v1.22.4   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
+ip-10-0-21-45   Ready    <none>   4m28s   v1.22.4   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-40-36   Ready    <none>   4m22s   v1.22.4   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
 ```
 

docs/advanced/customization.md

@@ -12,9 +12,9 @@ Clusters are kept to a minimal Kubernetes control plane by offering components l
 
 ## Hosts
 
-Typhoon uses the [Ignition](https://github.com/coreos/ignition) system of Fedora CoreOS and Flatcar Linux to immutably declare a system via first-boot disk provisioning. Fedora CoreOS uses a [Fedora CoreOS Config](https://docs.fedoraproject.org/en-US/fedora-coreos/fcct-config/) (FCC) and Flatcar Linux uses a [Container Linux Config](https://github.com/coreos/container-linux-config-transpiler/blob/master/doc/examples.md) (CLC). These define disk partitions, filesystems, systemd units, dropins, config files, mount units, raid arrays, and users.
+Typhoon uses the [Ignition](https://github.com/coreos/ignition) system of Fedora CoreOS and Flatcar Linux to immutably declare a system via first-boot disk provisioning. Fedora CoreOS uses a [Butane Config](https://coreos.github.io/butane/specs/) and Flatcar Linux uses a [Container Linux Config](https://github.com/coreos/container-linux-config-transpiler/blob/master/doc/examples.md) (CLC). These define disk partitions, filesystems, systemd units, dropins, config files, mount units, raid arrays, and users.
 
-Controller and worker instances form a minimal and secure Kubernetes cluster on each platform. Typhoon provides the **snippets** feature to accept Fedora CoreOS Configs or Container Linux Configs to validate and additively merge into instance declarations. This allows advanced host customization and experimentation.
+Controller and worker instances form a minimal and secure Kubernetes cluster on each platform. Typhoon provides the **snippets** feature to accept Butane or Container Linux Configs to validate and additively merge into instance declarations. This allows advanced host customization and experimentation.
 
 !!! note
     Snippets cannot be used to modify an already existing instance, the antithesis of immutable provisioning. Ignition fully declares a system on first boot only.
@@ -30,14 +30,14 @@ Controller and worker instances form a minimal and secure Kubernetes cluster on
 !!! note
     Fedora CoreOS snippets require `terraform-provider-ct` v0.5+
 
-Define a Fedora CoreOS Config (FCC) ([docs](https://docs.fedoraproject.org/en-US/fedora-coreos/fcct-config/), [config](https://github.com/coreos/fcct/blob/master/docs/configuration-v1_0.md), [examples](https://github.com/coreos/fcct/blob/master/docs/examples.md)) in version control near your Terraform workspace directory (e.g. perhaps in a `snippets` subdirectory). You may organize snippets into multiple files, if desired.
+Define a Butane Config ([docs](https://coreos.github.io/butane/specs/), [config](https://github.com/coreos/butane/blob/main/docs/config-fcos-v1_4.md)) in version control near your Terraform workspace directory (e.g. perhaps in a `snippets` subdirectory). You may organize snippets into multiple files, if desired.
 
 For example, ensure an `/opt/hello` file is created with permissions 0644.
 
 ```yaml
 # custom-files
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 storage:
   files:
     - path: /opt/hello
@@ -185,7 +185,7 @@ To set an alternative etcd image or Kubelet image, use a snippet to set a system
     ```yaml
     # kubelet-image-override.yaml
     variant: fcos           <- remove for Flatcar Linux
-    version: 1.1.0          <- remove for Flatcar Linux
+    version: 1.4.0          <- remove for Flatcar Linux
     systemd:
       units:
         - name: kubelet.service
@@ -201,7 +201,7 @@ To set an alternative etcd image or Kubelet image, use a snippet to set a system
     ```yaml
     # etcd-image-override.yaml
     variant: fcos           <- remove for Flatcar Linux
-    version: 1.1.0          <- remove for Flatcar Linux
+    version: 1.4.0          <- remove for Flatcar Linux
     systemd:
       units:
         - name: etcd-member.service

docs/advanced/nodes.md

@@ -0,0 +1,134 @@
+# Nodes
+
+Typhoon clusters consist of controller node(s) and a (default) set of worker nodes.
+
+## Overview
+
+Typhoon nodes use the standard set of Kubernetes node labels.
+
+```yaml
+Labels: kubernetes.io/arch=amd64
+        kubernetes.io/hostname=node-name
+        kubernetes.io/os=linux
+```
+
+Controller node(s) are labeled to allow node selection (for rare components that run on controllers) and tainted to prevent ordinary workloads running on controllers.
+
+```yaml
+Labels: node.kubernetes.io/controller=true
+Taints: node-role.kubernetes.io/controller:NoSchedule
+```
+
+Worker nodes are labeled to allow node selection and untainted. Workloads will schedule on worker nodes by default, baring any contraindications.
+
+```yaml
+Labels: node.kubernetes.io/node=
+Taints: <none>
+```
+
+On auto-scaling cloud platforms, you may add [worker pools](/advanced/worker-pools/) with different groups of nodes with their own labels and taints. On platforms like bare-metal, with heterogeneous machines, you may manage node labels and taints per node.
+
+## Node Labels
+
+Add custom initial worker node labels to default workers or worker pool nodes to allow workloads to select among nodes that differ.
+
+=== "Cluster"
+
+    ```tf
+    module "yavin" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.4"
+
+      # Google Cloud
+      cluster_name  = "yavin"
+      region        = "us-central1"
+      dns_zone      = "example.com"
+      dns_zone_name = "example-zone"
+
+      # configuration
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count = 2
+      worker_node_labels = ["pool=default"]
+    }
+    ```
+
+=== "Worker Pool"
+
+    ```tf
+    module "yavin-pool" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.4"
+
+      # Google Cloud
+      cluster_name = "yavin"
+      region       = "europe-west2"
+      network      = module.yavin.network_name
+
+      # configuration
+      name               = "yavin-16x"
+      kubeconfig         = module.yavin.kubeconfig
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count = 1
+      machine_type = "n1-standard-16"
+      node_labels  = ["pool=big"]
+    }
+    ```
+
+In the example above, the two default workers would be labeled `pool: default` and the additional worker would be labeled `pool: big`.
+
+## Node Taints
+
+Add custom initial taints on worker pool nodes to indicate a node is unique and should only schedule workloads that explicitly tolerate a given taint key.
+
+!!! warning
+    Since taints prevent workloads scheduling onto a node, you must decide whether `kube-system` DaemonSets (e.g. flannel, Calico, Cilium) should tolerate your custom taint by setting `daemonset_tolerations`. If you don't list your custom taint(s), important components won't run on these nodes.
+
+=== "Cluster"
+
+    ```tf
+    module "yavin" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.4"
+
+      # Google Cloud
+      cluster_name  = "yavin"
+      region        = "us-central1"
+      dns_zone      = "example.com"
+      dns_zone_name = "example-zone"
+
+      # configuration
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count = 2
+      daemonset_tolerations = ["role"]
+    }
+    ```
+
+=== "Worker Pool"
+
+    ```tf
+    module "yavin-pool" {
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.4"
+
+      # Google Cloud
+      cluster_name = "yavin"
+      region       = "europe-west2"
+      network      = module.yavin.network_name
+
+      # configuration
+      name               = "yavin-16x"
+      kubeconfig         = module.yavin.kubeconfig
+      ssh_authorized_key = local.ssh_key
+
+      # optional
+      worker_count      = 1
+      accelerator_type  = "nvidia-tesla-p100"
+      accelerator_count = 1
+      node_taints       = ["role=gpu:NoSchedule"]
+    }
+    ```
+
+In the example above, the the additional worker would be tainted with `role=gpu:NoSchedule` to prevent workloads scheduling, but `kube-system` components like flannel, Calico, or Cilium would tolerate that custom taint to run there.
+

docs/advanced/worker-pools.md

@@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.20.4"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.22.4"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.20.4"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.22.4"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -82,7 +82,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | subnet_ids | Must be set to `subnet_ids` output by cluster | module.cluster.subnet_ids |
 | security_groups | Must be set to `worker_security_groups` output by cluster | module.cluster.worker_security_groups |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 #### Optional
 
@@ -90,15 +90,16 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | instance_type | EC2 instance type | "t3.small" | "t3.medium" |
-| os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge |
+| os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
 | disk_size | Size of the EBS volume in GB | 40 | 100 |
-| disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| disk_type | Type of the EBS volume | "gp3" | standard, gp2, gp3, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | spot_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0 | 0.10 |
 | snippets | Fedora CoreOS or Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |
+| node_taints | List of initial node taints | [] | ["role=gpu:NoSchedule"] |
 
 Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/) or per-region and per-type [spot prices](https://aws.amazon.com/ec2/spot/pricing/).
 
@@ -110,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.20.4"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.22.4"
 
       # Azure
       region                  = module.ramius.region
@@ -136,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.20.4"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.22.4"
 
       # Azure
       region                  = module.ramius.region
@@ -181,7 +182,7 @@ The Azure internal `workers` module supports a number of [variables](https://git
 | security_group_id | Must be set to `security_group_id` output by cluster | module.cluster.security_group_id |
 | backend_address_pool_id | Must be set to `backend_address_pool_id` output by cluster | module.cluster.backend_address_pool_id |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 #### Optional
 
@@ -189,11 +190,12 @@ The Azure internal `workers` module supports a number of [variables](https://git
 |:-----|:------------|:--------|:--------|
 | worker_count | Number of instances | 1 | 3 |
 | vm_type | Machine type for instances | "Standard_DS1_v2" | See below |
-| os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge |
+| os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
 | priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | "Regular" | "Spot" |
 | snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |
+| node_taints | List of initial node taints | [] | ["role=gpu:NoSchedule"] |
 
 Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
 
@@ -205,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.20.4"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.4"
 
       # Google Cloud
       region       = "europe-west2"
@@ -229,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.20.4"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.22.4"
 
       # Google Cloud
       region       = "europe-west2"
@@ -260,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.20.4
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.20.4
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.20.4
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.20.4
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.20.4
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.22.4
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.22.4
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.22.4
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.22.4
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.22.4
 ```
 
 ### Variables
@@ -281,7 +283,7 @@ The Google Cloud internal `workers` module supports a number of [variables](http
 | network | Must be set to `network_name` output by cluster | module.cluster.network_name |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
 | os_image | Container Linux image for compute instances | "uploaded-flatcar-image" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`.
 
@@ -297,6 +299,7 @@ Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-z
 | snippets | Container Linux Config snippets | [] | [examples](/advanced/customization/) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |
+| node_taints | List of initial node taints | [] | ["role=gpu:NoSchedule"] |
 
 Check the list of valid [machine types](https://cloud.google.com/compute/docs/machine-types).
 

docs/architecture/operating-systems.md

@@ -16,10 +16,10 @@ Together, they diversify Typhoon to support a range of container technologies.
 
 | Property          | Flatcar Linux | Fedora CoreOS |
 |-------------------|---------------------------------|---------------|
-| Kernel            | ~5.4.x | ~5.8.x |
-| systemd           | 245 | 245 |
+| Kernel            | ~5.10.x | ~5.12.x |
+| systemd           | 247 | 248 |
 | Ignition system   | Ignition v2.x spec | Ignition v3.x spec |
-| Container Engine  | docker 19.3.12  | docker 19.03.11 |
+| Container Engine  | docker 19.3.15  | docker 20.10.6 |
 | storage driver    | overlay2 (extfs)  | overlay2 (xfs) |
 | logging driver    | json-file | journald |
 | cgroup driver     | cgroupfs (except Flatcar edge) | systemd  |
@@ -46,7 +46,7 @@ Typhoon conventional directories.
 
 | Kubelet setting   | Host location                  |
 |-------------------|--------------------------------|
-| cni-conf-dir      | /etc/kubernetes/cni/net.d      |
+| cni-conf-dir      | /etc/cni/net.d                 |
 | pod-manifest-path | /etc/kubernetes/manifests      |
 | volume-plugin-dir | /var/lib/kubelet/volumeplugins |
 

docs/fedora-coreos/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.20.4 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.22.4 cluster on AWS with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -51,11 +51,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.9.0"
     }
     aws = {
       source = "hashicorp/aws"
-      version = "3.28.0"
+      version = "3.48.0"
     }
   }
 }
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.20.4"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.4"
 
   # AWS
   cluster_name = "tempest"
@@ -80,7 +80,7 @@ module "tempest" {
   dns_zone_id  = "Z3PAABBCFAKEC0"
 
   # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
   # optional
   worker_count = 2
@@ -95,7 +95,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
 
 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```
 
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.20.4
-ip-10-0-26-65  Ready   <none>   10m  v1.20.4
-ip-10-0-41-21  Ready   <none>   10m  v1.20.4
+ip-10-0-3-155  Ready   <none>   10m  v1.22.4
+ip-10-0-26-65  Ready   <none>   10m  v1.22.4
+ip-10-0-41-21  Ready   <none>   10m  v1.22.4
 ```
 
 List the pods.
@@ -183,7 +183,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/aws/fed
 | cluster_name | Unique cluster name (prepended to dns_zone) | "tempest" |
 | dns_zone | AWS Route53 DNS zone | "aws.example.com" |
 | dns_zone_id | AWS Route53 DNS zone id | "Z3PAABBCFAKEC0" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 #### DNS Zone
 
@@ -211,13 +211,13 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
-| disk_size | Size of the EBS volume in GB | 40 | 100 |
-| disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| disk_size | Size of the EBS volume in GB | 30 | 100 |
+| disk_type | Type of the EBS volume | "gp3" | standard, gp2, gp3, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | worker_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0 | 0.10 |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
+| controller_snippets | Controller Butane snippets | [] | [examples](/advanced/customization/) |
+| worker_snippets | Worker Butane snippets | [] | [examples](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
 | host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |

docs/fedora-coreos/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.20.4 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.22.4 cluster on Azure with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -48,11 +48,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.9.0"
     }
     azurerm = {
       source = "hashicorp/azurerm"
-      version = "2.48.0"
+      version = "2.68.0"
     }
   }
 }
@@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.20.4"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.22.4"
 
   # Azure
   cluster_name   = "ramius"
@@ -96,7 +96,7 @@ module "ramius" {
 
   # configuration
   os_image           = "/subscriptions/some/path/Microsoft.Compute/images/fedora-coreos-31.20200323.3.2"
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
   # optional
   worker_count    = 2
@@ -111,7 +111,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
 
 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```
 
@@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.20.4
-ramius-worker-000001  Ready   <none>  25m  v1.20.4
-ramius-worker-000002  Ready   <none>  24m  v1.20.4
+ramius-controller-0   Ready   <none>  24m  v1.22.4
+ramius-worker-000001  Ready   <none>  25m  v1.22.4
+ramius-worker-000002  Ready   <none>  24m  v1.22.4
 ```
 
 List the pods.
@@ -201,7 +201,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/f
 | dns_zone | Azure DNS zone | "azure.example.com" |
 | dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
 | os_image | Fedora CoreOS image for instances | "/subscriptions/..../custom-image" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 !!! tip
     Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
@@ -241,10 +241,10 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "Standard_B2s" | See below |
 | worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
-| disk_size | Size of the disk in GB | 40 | 100 |
+| disk_size | Size of the disk in GB | 30 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
+| controller_snippets | Controller Butane snippets | [] | [example](/advanced/customization/#usage) |
+| worker_snippets | Worker Butane snippets | [] | [example](/advanced/customization/#usage) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |

docs/fedora-coreos/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.20.4 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.22.4 cluster on bare-metal with Fedora CoreOS.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -111,7 +111,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -138,11 +138,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.9.0"
     }
     matchbox = {
       source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
     }
   }
 }
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.20.4"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.22.4"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -164,7 +164,7 @@ module "mercury" {
 
   # configuration
   k8s_domain_name    = "node1.example.com"
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
   # machines
   controllers = [{
@@ -194,7 +194,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
 
 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```
 
@@ -283,9 +283,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.20.4
-node2.example.com   Ready   <none>  10m  v1.20.4
-node3.example.com   Ready   <none>  10m  v1.20.4
+node1.example.com   Ready   <none>  10m  v1.22.4
+node2.example.com   Ready   <none>  10m  v1.22.4
+node3.example.com   Ready   <none>  10m  v1.22.4
 ```
 
 List the pods.
@@ -323,7 +323,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | os_stream | Fedora CoreOS release stream | "stable" |
 | os_version | Fedora CoreOS version to PXE and install | "32.20201104.3.0" |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3Nz..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3Nz..." |
 | controllers | List of controller machine detail objects (unique name, identifying MAC address, FQDN) | `[{name="node1", mac="52:54:00:a1:9c:ae", domain="node1.example.com"}]` |
 | workers | List of worker machine detail objects (unique name, identifying MAC address, FQDN) | `[{name="node2", mac="52:54:00:b2:2f:86", domain="node2.example.com"}, {name="node3", mac="52:54:00:c3:61:77", domain="node3.example.com"}]` |
 
@@ -335,7 +335,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | install_disk | Disk device where Fedora CoreOS should be installed | "sda" (not "/dev/sda" like Container Linux) | "sdb" |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | network_mtu | CNI interface MTU (calico-only) | 1480 | - |
-| snippets | Map from machine names to lists of Fedora CoreOS Config snippets | {} | [examples](/advanced/customization/) |
+| snippets | Map from machine names to lists of Butane snippets | {} | [examples](/advanced/customization/) |
 | network_ip_autodetection_method | Method to detect host IPv4 address (calico-only) | "first-found" | "can-reach=10.0.0.1" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |

docs/fedora-coreos/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.20.4 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.22.4 cluster on DigitalOcean with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -29,7 +29,7 @@ cd infra/clusters
 
 ## Provider
 
-Login to [DigitalOcean](https://cloud.digitalocean.com) or create an [account](https://cloud.digitalocean.com/registrations/new), if you don't have one.
+Login to [DigitalOcean](https://cloud.digitalocean.com). Or if you don't have one, create an account with our [referral link](https://m.do.co/c/94a5a4e76387) to get free credits.
 
 Generate a Personal Access Token with read/write scope from the [API tab](https://cloud.digitalocean.com/settings/api/tokens). Write the token to a file that can be referenced in configs.
 
@@ -51,7 +51,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.9.0"
     }
     digitalocean = {
       source = "digitalocean/digitalocean"
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.20.4"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.22.4"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -104,7 +104,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
 
 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```
 
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.20.4
-10.132.115.81      Ready   <none>  10m  v1.20.4
-10.132.124.107     Ready   <none>  10m  v1.20.4
+10.132.110.130     Ready   <none>  10m  v1.22.4
+10.132.115.81      Ready   <none>  10m  v1.22.4
+10.132.124.107     Ready   <none>  10m  v1.22.4
 ```
 
 List the pods.
@@ -214,7 +214,7 @@ resource "digitalocean_domain" "zone-for-clusters" {
 
 #### SSH Fingerprints
 
-DigitalOcean droplets are created with your SSH public key "fingerprint" (i.e. MD5 hash) to allow access. If your SSH public key is at `~/.ssh/id_rsa`, find the fingerprint with,
+DigitalOcean droplets are created with your SSH public key "fingerprint" (i.e. MD5 hash) to allow access. If your SSH public key is at `~/.ssh/id_ed25519.pub`, find the fingerprint with,
 
 ```bash
 ssh-keygen -E md5 -lf ~/.ssh/id_ed25519.pub | awk '{print $2}'
@@ -238,8 +238,8 @@ Digital Ocean requires the SSH public key be uploaded to your account, so you ma
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Droplet type for controllers | "s-2vcpu-2gb" | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
 | worker_type | Droplet type for workers | "s-1vcpu-2gb" | s-1vcpu-2gb, s-2vcpu-2gb, ... |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/) |
+| controller_snippets | Controller Butane snippets | [] | [example](/advanced/customization/) |
+| worker_snippets | Worker Butane snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |

docs/fedora-coreos/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.20.4 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.22.4 cluster on Google Compute Engine with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -52,11 +52,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.9.0"
     }
     google = {
       source = "hashicorp/google"
-      version = "3.57.0"
+      version = "3.75.0"
     }
   }
 }
@@ -82,7 +82,7 @@ module "yavin" {
   dns_zone_name = "example-zone"
 
   # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 
   # optional
   worker_count = 2
@@ -96,7 +96,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
 
 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```
 
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.4
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.4
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.4
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.4
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.4
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.4
 ```
 
 List the pods.
@@ -186,7 +186,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-
 | region | Google Cloud region | "us-central1" |
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Fedora CoreOS [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep fedora-coreos`.
 
@@ -216,10 +216,10 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "stable", "testing", "next" |
-| disk_size | Size of the disk in GB | 40 | 100 |
+| disk_size | Size of the disk in GB | 30 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
+| controller_snippets | Controller Butane snippets | [] | [examples](/advanced/customization/) |
+| worker_snippets | Worker Butane snippets | [] | [examples](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |

docs/flatcar-linux/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.20.4 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.22.4 cluster on AWS with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.13.0+ on your sy
 
 ```sh
 $ terraform version
-Terraform v0.13.0
+Terraform v1.0.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -51,11 +51,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.7.1"
+      version = "0.9.0"
     }
     aws = {
       source = "hashicorp/aws"
-      version = "3.28.0"
+      version = "3.48.0"
     }
   }
 }
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.20.4"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.22.4"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.20.4
-ip-10-0-26-65  Ready   <none>  10m  v1.20.4
-ip-10-0-41-21  Ready   <none>  10m  v1.20.4
+ip-10-0-3-155  Ready   <none>  10m  v1.22.4
+ip-10-0-26-65  Ready   <none>  10m  v1.22.4
+ip-10-0-41-21  Ready   <none>  10m  v1.22.4
 ```
 
 List the pods.
@@ -210,9 +210,9 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
-| os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge |
-| disk_size | Size of the EBS volume in GB | 40 | 100 |
-| disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
+| disk_size | Size of the EBS volume in GB | 30 | 100 |
+| disk_type | Type of the EBS volume | "gp3" | standard, gp2, gp3, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | worker_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0/null | 0.10 |