Update Terraform provider recommendations in docs

Update nginx-ingress and Grafana addons
Upgrade to DigitalOcean Terraform provider v2.x
2025-08-02 17:51:33 +02:00 · 2022-01-19 21:16:37 -08:00 · 2022-01-19 21:09:21 -08:00 · 2022-01-19 18:32:17 -08:00 · 2022-01-19 17:59:49 -08:00 · 2022-01-19 16:42:21 -08:00
147 changed files with 1700 additions and 812 deletions
--- a/.github/FUNDING.yml
+++ b/.github/FUNDING.yml
@ -0,0 +1 @@
+github: [poseidon]
--- a/.github/dependabot.yaml
+++ b/.github/dependabot.yaml
@ -3,7 +3,7 @@ updates:
 - package-ecosystem: pip
  directory: "/"
  schedule:
-    interval: daily
+    interval: weekly
  pull-request-branch-name:
    separator: "-"
  open-pull-requests-limit: 3
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,299 @@ Notable changes between versions.

 ## Latest

+* Kubernetes [v1.23.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1232)
+* Remove Kubelet flag `--network-plugin`. Unused since `docker-shim` isn't used ([#1106](https://github.com/poseidon/typhoon/pull/1106))
+
+### Fedora CoreOS
+
+* Switch Kubernetes Container Runtime from `docker` to `containerd` ([#1101](https://github.com/poseidon/typhoon/pull/1101))
+* Mask `docker.service` to prevent it from being socket activated ([#1105](https://github.com/poseidon/typhoon/pull/1105))
+
+### Flatcar Linux
+
+#### AWS
+
+* Add experimental Flatcar Linux ARM64 support ([docs](https://typhoon.psdn.io/advanced/arm64/), [#1102](https://github.com/poseidon/typhoon/pull/1102))
+  * Add `arch` variable to AWS `kubernetes` and `workers` modules
+  * Allow arm64 full-cluster or mixed/hybrid cluster with arm64 workers
+  * Requires `flannel` or `cilium` CNI provider
+
+### DigitalOcean
+
+* Upgrade DigitalOcean Terraform provider to [v2.x](https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs) ([#1109](https://github.com/poseidon/typhoon/pull/1109))
+
+### Addons
+
+* Update nginx-ingress from v1.1.0 to [v1.1.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.1)
+* Update Grafana from v8.3.3 to [v8.3.4](https://github.com/grafana/grafana/releases/tag/v8.3.4)
+
+## v1.23.1
+
+* Kubernetes [v1.23.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1231)
+* Workaround Terraform v1.1 regression in `file` provisioner ([#1093](https://github.com/poseidon/typhoon/pull/1093))
+
+### Flatcar Linux
+
+* Switch Kubernetes Container Runtime from `docker` to `containerd` ([#1087](https://github.com/poseidon/typhoon/pull/1087))
+
+### Addons
+
+* Configure Prometheus to allow a custom scrape query parameter ([#1095](https://github.com/poseidon/typhoon/pull/1095))
+* Configure Prometheus to probe Kubernetes Ingress via `blackbox-exporter` ([#1096](https://github.com/poseidon/typhoon/pull/1096))
+* Fix Prometheus Service probes to use `blackbox-exporter`, not `blackbox` ([#1096](https://github.com/poseidon/typhoon/pull/1096))
+
+## v1.23.0
+
+* Kubernetes [v1.23.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230)
+* Normalize CA cert mounts in static Pods and kube-proxy ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+* Set Kubelet resolver config to `/run/systemd/resolve/resolv.conf` ([#1082](https://github.com/poseidon/typhoon/pull/1082))
+* Update Cilium from v1.10.5 to [v1.11.0](https://github.com/cilium/cilium/releases/tag/v1.11.0) ([#1083](https://github.com/poseidon/typhoon/pull/1083))
+* With Calico, add missing `caliconodestatuses` CRD ([#289](https://github.com/poseidon/terraform-render-bootstrap/pull/289))
+* Change `enable_aggregation` default to true ([#279](https://github.com/poseidon/terraform-render-bootstrap/pull/279))
+* Remove deprecated `--port` from `kube-scheduler` ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+
+### AWS
+
+* Change controller node default `disk_iops` to 3000 ([#1073](https://github.com/poseidon/typhoon/pull/1073))
+
+### Azure
+
+* Fix warning about deprecated `backend_address_pool_id` ([#1086](https://github.com/poseidon/typhoon/pull/1086))
+
+### Fedora CoreOS
+
+* Fix Fedora ARM64 workers to official Fedora CoreOS AMIs ([#1072](https://github.com/poseidon/typhoon/pull/1072))
+  * Should have been changed alongside controller AMIs in ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+  * Old Poseidon built ARM64 AMIs have been deleted
+
+### Addons
+
+* Update nginx-ingress from v1.0.5 to [v1.1.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.0)
+* Update Prometheus from v2.31.1 to [v2.32.0](https://github.com/prometheus/prometheus/releases/tag/v2.32.0)
+* Update kube-state-metrics from v2.2.4 to [v2.3.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.3.0)
+* Update node-exporter from v1.3.0 to [v1.3.1](https://github.com/prometheus/node_exporter/releases/tag/v1.3.1)
+* Update Grafana from v8.2.4 to [v8.3.3](https://github.com/grafana/grafana/releases/tag/v8.3.3)
+
+### Known Issues
+
+* Calico does not yet support Kubernetes v1.23.0, use `flannel` or `cilium` ([calico#5011](https://github.com/projectcalico/calico/issues/5011))
+
+## v1.22.4
+
+* Kubernetes [v1.22.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224)
+* Update CoreDNS from v1.8.4 to [v1.8.6](https://github.com/poseidon/terraform-render-bootstrap/pull/284)
+* Update Calico from v3.20.2 to [v3.21.0](https://github.com/projectcalico/calico/releases/tag/v3.21.0)
+* Update flannel from v0.14.0 to [v0.15.1](https://github.com/flannel-io/flannel/releases/tag/v0.15.1)
+
+### Google
+
+* Allow use of Terraform provider `google` [v4.0+](https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0)
+
+### Flatcar Linux
+
+* Change Kubelet mounts for cgroups v2 ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+* Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+
+### Addons
+
+* Update Prometheus from v2.30.3 to [v2.31.1](https://github.com/prometheus/prometheus/releases/tag/v2.31.1)
+* Update node-exporter from v1.2.2 to [v1.3.0](https://github.com/prometheus/node_exporter/releases/tag/v1.3.0)
+* Update kube-state-metrics from v2.2.3 to [v2.2.4](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.4)
+* Update Grafana from v8.2.1 to [v8.2.4](https://github.com/grafana/grafana/releases/tag/v8.2.4)
+* Update nginx-ingress from v1.0.4 to [v1.0.5](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.5)
+
+## v1.23.3
+
+* Kubernetes [v1.22.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223)
+* Update etcd from v3.5.0 to [v3.5.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.1)
+* Update Cilium from v1.10.4 to [v1.10.5](https://github.com/cilium/cilium/releases/tag/v1.10.5)
+* Update Calico from v3.20.1 to [v3.20.2](https://github.com/projectcalico/calico/releases/tag/v3.20.2)
+  * Use Calico's iptables legacy vs nft auto-detection
+* Update flannel from v0.13.0 to v0.14.0
+
+### Bare-Metal
+
+* Require Terraform provider `poseidon/matchbox` v0.5+ ([#1048](https://github.com/poseidon/typhoon/pull/1048))
+
+### Addons
+
+* Update nginx-ingress from v1.0.0 to [v1.0.4](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4)
+* Update Prometheus from v2.29.2 to [v2.30.3](https://github.com/prometheus/prometheus/releases/tag/v2.30.3)
+* Update kube-state-metrics from v2.2.0 to [v2.2.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.3)
+* Update Grafana from v8.1.2 to [v8.2.1](https://github.com/grafana/grafana/releases/tag/v8.2.1)
+
+## v1.22.2
+
+* Kubernetes [v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222)
+* Update Cilium from v1.10.3 to [v1.10.4](https://github.com/cilium/cilium/releases/tag/v1.10.4)
+* Update Calico from v3.20.0 to [v3.20.1](https://github.com/projectcalico/calico/releases/tag/v3.20.1)
+* Fix access to ClusterIP services with Cilium ([#276](https://github.com/poseidon/terraform-render-bootstrap/pull/276))
+
+### Fedora CoreOS
+
+* Use Fedora CoreOS ARM64 AMIs ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+
+### Addons
+
+* Update Prometheus from v2.29.1 to [v2.29.2](https://github.com/prometheus/prometheus/releases/tag/v2.29.2)
+* Update kube-state-metrics from v2.1.1 to [v2.2.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.0)
+
+## v1.22.1
+
+* Kubernetes [v1.22.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221)
+* Update Calico from v3.19.1 to [v3.20.0](https://github.com/projectcalico/calico/releases/tag/v3.20.0)
+
+### Addons
+
+* Update nginx-ingress from v1.0.0-beta.1 to [v1.0.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0)
+* Update Prometheus from v2.28.1 to [v2.29.1](https://github.com/prometheus/prometheus/releases/tag/v2.29.1)
+* Update Grafana from v8.1.1 to [v8.1.2](https://github.com/grafana/grafana/releases/tag/v8.1.2)
+
+## v1.22.0
+
+* Kubernetes [v1.22.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1220)
+* Update etcd from v3.4.16 to [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0)
+* Switch `kube-controller-manager` and `kube-scheduler` to use secure port only
+  * Update Prometheus config to discover endpoints and use a bearer token to scrape
+
+### Fedora CoreOS
+
+* Add Cilium cgroups v2 support on Fedora CoreOS
+* Update Butane Config version from v1.2.0 to v1.4.0
+  * Rename Fedora CoreOS Config to Butane Config
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.4.0
+
+### Addons
+
+* Update nginx-ingress from v0.47.0 to [v1.0.0-beta.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0-beta.1)
+* Update node-exporter from v1.2.0 to [v1.2.2](https://github.com/prometheus/node_exporter/releases/tag/v1.2.2)
+* Update kube-state-metrics from v2.1.0 to [v2.1.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.1)
+* Update Grafana from v8.0.6 to [v8.1.1](https://github.com/grafana/grafana/releases/tag/v8.1.1)
+
+## v1.21.3
+
+* Kubernetes [v1.21.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1213)
+* Update Cilium from v1.10.1 to [v1.10.3](https://github.com/cilium/cilium/releases/tag/v1.10.3)
+* Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.9+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+
+### AWS
+
+* Change default disk type from `gp2` to `gp3` ([#1012](https://github.com/poseidon/typhoon/pull/1012))
+
+### Addons
+
+* Update Prometheus from v2.28.0 to [v2.28.1](https://github.com/prometheus/prometheus/releases/tag/v2.28.1)
+* Update node-exporter from v1.1.2 to [v1.2.0](https://github.com/prometheus/node_exporter/releases/tag/v1.2.0)
+* Update Grafana from v8.0.3 to [v8.0.6](https://github.com/grafana/grafana/releases/tag/v8.0.6)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
+## v1.21.2
+
+* Kubernetes [v1.21.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1212)
+* Add Terraform v1.0.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x, v0.14.4+, and v0.15.x
+* Update CoreDNS from v1.8.0 to [v1.8.4]([#1006](https://github.com/poseidon/typhoon/pull/1006))
+* Update Cilium from v1.9.6 to [v1.10.1](https://github.com/cilium/cilium/releases/tag/v1.10.1)
+* Update Calico from v3.19.0 to [v3.19.1](https://github.com/projectcalico/calico/releases/tag/v3.19.1)
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0 to [v2.1.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.0)
+* Update Prometheus from v2.27.0 to [v2.28.0](https://github.com/prometheus/prometheus/releases/tag/v2.28.0)
+* Update Grafana from v7.5.6 to [v8.0.3](https://github.com/grafana/grafana/releases/tag/v8.0.3)
+* Update nginx-ingress from v0.46.0 to [v0.47.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.47.0)
+
+### Fedora CoreOS
+
+#### AWS
+
+* Extend experimental Fedora CoreOS arm64 support with Cilium
+  * CNI provider may now be `flannel` or `cilium` (new)
+
+#### Bare-Metal
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+#### DigitalOcean
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
+## v1.21.1
+
+* Kubernetes [v1.21.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211)
+* Add Terraform v0.15.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x and v0.14.4+
+* Update etcd from v3.4.15 to [v3.4.16](https://github.com/etcd-io/etcd/releases/tag/v3.4.16)
+* Update Cilium from v1.9.5 to [v1.9.6](https://github.com/cilium/cilium/releases/tag/v1.9.6)
+* Update Calico from v3.18.1 to [v3.19.0](https://github.com/projectcalico/calico/releases/tag/v3.19.0)
+
+### AWS
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Azure
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Google Cloud
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Fedora CoreOS
+
+* Update Kubelet mounts for cgroups v2 ([#978](https://github.com/poseidon/typhoon/pull/978))
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0-rc.1 to [v2.0.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0)
+* Update Prometheus from v2.25.2 to [v2.27.0](https://github.com/prometheus/prometheus/releases/tag/v2.27.0)
+* Update Grafana from v7.5.3 to [v7.5.6](https://github.com/grafana/grafana/releases/tag/v7.5.6)
+* Update nginx-ingress from v0.45.0 to [v0.46.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.46.0)
+
+## v1.21.0
+
+* Kubernetes [v1.21.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1210)
+  * Enable `tokencleaner` controller ([#969](https://github.com/poseidon/typhoon/pull/969))
+  * Enable `kube-scheduler` and `kube-controller-manager` separate authn/z kubeconfig
+  * Change CNI config location from /etc/kubernetes/cni/net.d to /etc/cni/net.d ([#965](https://github.com/poseidon/typhoon/pull/965))
+  * Change `kube-controller-manager` to mount `/var/lib/kubelet/volumeplugins` directly
+  * Remove unused `cloud-provider` flags
+* Update Fedora CoreOS Config version from v1.1.0 to v1.2.0 ([#970](https://github.com/poseidon/typhoon/pull/970))
+  * Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.8+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.2.0
+
+### AWS
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Azure
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+* Remove deprecated `azurerm_lb_backend_address_pool` field `resource_group_name` ([#972](https://github.com/poseidon/typhoon/pull/972))
+
+### Google Cloud
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Addons
+
+* Update nginx-ingress from v0.44.0 to [v0.45.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.45.0)
+* Update kube-state-metrics from v2.0.0-rc.0 to [v2.0.0-rc.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-rc.1)
+* Update Grafana from v7.4.5 to [v7.5.3](https://github.com/grafana/grafana/releases/tag/v7.5.3)
+
 ## v1.20.5

 * Kubernetes [v1.20.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1205)
--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@ -31,6 +31,10 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | stable |

+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora CoreOS (ARM64) | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | alpha |
+
 Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).

 | Platform      | Operating System | Terraform Module | Status |
@ -41,6 +45,10 @@ Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/
 | DigitalOcean | Flatcar Linux  | [digital-ocean/flatcar-linux/kubernetes](digital-ocean/flatcar-linux/kubernetes) | beta |
 | Google Cloud  | Flatcar Linux  | [google-cloud/flatcar-linux/kubernetes](google-cloud/flatcar-linux/kubernetes) | beta |

+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Flatcar Linux (ARM64) | [aws/flatcar-linux/kubernetes](aws/flatcar-linux/kubernetes) | alpha |
+
 ## Documentation

 * [Docs](https://typhoon.psdn.io)
@ -54,7 +62,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.20.5"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.2"

  # Google Cloud
  cluster_name  = "yavin"
@ -63,7 +71,7 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count = 2
@ -93,9 +101,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.20.5
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.20.5
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.20.5
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.2
 ```

 List the pods.
@ -126,7 +134,7 @@ Typhoon is strict about minimalism, maturity, and scope. These are not in scope:

 ## Help

-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
+Schedule a meeting via [Github Sponsors](https://github.com/sponsors/poseidon?frequency=one-time) to discuss your use case.

 ## Motivation

--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -24,7 +24,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:7.4.5
+          image: docker.io/grafana/grafana:8.3.4
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.44.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/prometheus/config.yaml
+++ b/addons/prometheus/config.yaml
@ -72,6 +72,48 @@ data:
        regex: apiserver_request_duration_seconds_count;.+
        action: drop

+    # Scrape config for kube-controller-manager endpoints.
+    #
+    # kube-controller-manager service endpoints can be discovered by using the
+    # `endpoints` role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-controller-manager and the `https` port.
+    - job_name: 'kube-controller-manager'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-controller-manager;metrics
+      - replacement: kube-controller-manager
+        action: replace
+        target_label: job
+
+    # Scrape config for kube-scheduler endpoints.
+    #
+    # kube-scheduler service endpoints can be discovered by using the `endpoints`
+    # role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-scheduler and the `https` port.
+    - job_name: 'kube-scheduler'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-scheduler;metrics
+      - replacement: kube-scheduler
+        action: replace
+        target_label: job
+
    # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
    # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
    - job_name: 'kubelet'
@ -133,6 +175,7 @@ data:
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
+    # * `prometheus.io/param`: Custom metrics query parameter, like "format=prometheus".
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
@ -155,6 +198,11 @@ data:
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_param]
+        action: replace
+        target_label: __param_$1
+        regex: ([^=]+)=(.*)
+        replacement: $2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
@ -172,38 +220,6 @@ data:
        action: drop
        regex: etcd_(debugging|disk|request|server).*

-    # Example scrape config for probing services via the Blackbox Exporter.
-    #
-    # The relabeling allows the actual service scrape endpoint to be configured
-    # via the following annotations:
-    #
-    # * `prometheus.io/probe`: Only probe services that have a value of `true`
-    - job_name: 'kubernetes-services'
-
-      metrics_path: /probe
-      params:
-        module: [http_2xx]
-
-      kubernetes_sd_configs:
-      - role: service
-
-      relabel_configs:
-      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
-        action: keep
-        regex: true
-      - source_labels: [__address__]
-        target_label: __param_target
-      - target_label: __address__
-        replacement: blackbox
-      - source_labels: [__param_target]
-        target_label: instance
-      - action: labelmap
-        regex: __meta_kubernetes_service_label_(.+)
-      - source_labels: [__meta_kubernetes_namespace]
-        target_label: namespace
-      - source_labels: [__meta_kubernetes_service_name]
-        target_label: job
-
    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
@ -240,6 +256,67 @@ data:
        action: replace
        target_label: kubernetes_pod_name

+    # Example scrape config for probing Services via the Blackbox Exporter.
+    #
+    # Relabeling allows service scraping to be configured via annotations:
+    # * `prometheus.io/probe`: Only probe services that have a value of `true`
+    - job_name: 'kubernetes-services'
+
+      metrics_path: /probe
+      params:
+        module: [http_2xx]
+
+      kubernetes_sd_configs:
+      - role: service
+
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
+        action: keep
+        regex: true
+      - source_labels: [__address__]
+        target_label: __param_target
+      - target_label: __address__
+        replacement: blackbox-exporter:8080
+      - source_labels: [__param_target]
+        target_label: instance
+      - action: labelmap
+        regex: __meta_kubernetes_service_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_service_name]
+        target_label: job
+
+    # Example scrape config for probing Ingresses via a Blackbox Exporter.
+    #
+    # Relabeling allows service scraping to be configured via annotations:
+    # * `prometheus.io/probe`: Only probe ingresses that have a value of `true`
+    - job_name: 'kubernetes-ingresses'
+      metrics_path: /probe
+      params:
+        module: [http_2xx]
+
+      kubernetes_sd_configs:
+      - role: ingress
+
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
+        regex: (.+);(.+);(.+)
+        replacement: ${1}://${2}${3}
+        target_label: __param_target
+      - target_label: __address__
+        replacement: blackbox-exporter:8080
+      - source_labels: [__param_target]
+        target_label: instance
+      - action: labelmap
+        regex: __meta_kubernetes_ingress_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_service_name]
+        target_label: job
+
    # Rule files
    rule_files:
      - "/etc/prometheus/rules/*.rules"
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -21,7 +21,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.25.2
+          image: quay.io/prometheus/prometheus:v2.32.1
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/discovery/kube-controller-manager.yaml
+++ b/addons/prometheus/discovery/kube-controller-manager.yaml
@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
  name: kube-controller-manager
  namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
  clusterIP: None
@ -14,5 +12,5 @@ spec:
  ports:
    - name: metrics
      protocol: TCP
-      port: 10252
-      targetPort: 10252
+      port: 10257
+      targetPort: 10257
--- a/addons/prometheus/discovery/kube-scheduler.yaml
+++ b/addons/prometheus/discovery/kube-scheduler.yaml
@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
  name: kube-scheduler
  namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
  clusterIP: None
@ -14,5 +12,5 @@ spec:
  ports:
    - name: metrics
      protocol: TCP
-      port: 10251
-      targetPort: 10251
+      port: 10259
+      targetPort: 10259
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -25,7 +25,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-rc.0
+        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -28,13 +28,13 @@ spec:
      hostPID: true
      containers:
      - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.1.2
+        image: quay.io/prometheus/node-exporter:v1.3.1
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
          - --path.rootfs=/host/root
-          - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
-          - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
+          - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)($|/)
+          - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
        ports:
          - name: metrics
            containerPort: 9100
--- a/addons/prometheus/rbac/cluster-role.yaml
+++ b/addons/prometheus/rbac/cluster-role.yaml
@ -10,6 +10,17 @@ rules:
  - services
  - endpoints
  - pods
-  verbs: ["get", "list", "watch"]
+  verbs:
+  - get
+  - list
+  - watch
 - nonResourceURLs: ["/metrics"]
  verbs: ["get"]
+- apiGroups:
+  - networking.k8s.io
+  resources:
+  - ingresses
+  verbs:
+  - get
+  - list
+  - watch
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
  }
 }

-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
  count = var.arch == "arm64" ? 1 : 0

  most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]

  filter {
    name   = "architecture"
@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
  }

  filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
-
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -13,7 +13,5 @@ module "bootstrap" {
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
  daemonset_tolerations = var.daemonset_tolerations
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/aws/fedora-coreos/kubernetes/controllers.tf
+++ b/aws/fedora-coreos/kubernetes/controllers.tf
@ -62,7 +62,6 @@ data "template_file" "controller-configs" {

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
-    etcd_arch   = var.arch == "arm64" ? "-arm64" : ""
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15${etcd_arch}
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -29,8 +29,10 @@ systemd:
        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -54,9 +56,9 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -67,14 +69,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -86,18 +88,19 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -123,7 +126,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.23.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -218,7 +221,26 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
-          ETCD_UNSUPPORTED_ARCH=arm64
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/security.tf
+++ b/aws/fedora-coreos/kubernetes/security.tf
@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
  source_security_group_id = aws_security_group.worker.id
 }

@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
  source_security_group_id = aws_security_group.worker.id
 }

--- a/aws/fedora-coreos/kubernetes/ssh.tf
+++ b/aws/fedora-coreos/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -55,19 +55,19 @@ variable "os_stream" {
 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -84,13 +84,13 @@ variable "worker_target_groups" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -142,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -176,4 +176,3 @@ variable "daemonset_tolerations" {
  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
  default     = []
 }
-
--- a/aws/fedora-coreos/kubernetes/versions.tf
+++ b/aws/fedora-coreos/kubernetes/versions.tf
@ -1,15 +1,15 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
  }
 }

-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
  count = var.arch == "arm64" ? 1 : 0

  most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]

  filter {
    name   = "architecture"
@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
  }

  filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
-
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -1,10 +1,12 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -27,9 +29,9 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -40,14 +42,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -59,14 +61,14 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
@ -77,6 +79,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -91,7 +94,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -130,9 +133,28 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ${ssh_authorized_key}
-
--- a/aws/fedora-coreos/kubernetes/workers/variables.tf
+++ b/aws/fedora-coreos/kubernetes/workers/variables.tf
@ -48,13 +48,13 @@ variable "os_stream" {
 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
@ -77,7 +77,7 @@ variable "target_groups" {

 variable "snippets" {
  type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
  default     = []
 }

--- a/aws/fedora-coreos/kubernetes/workers/versions.tf
+++ b/aws/fedora-coreos/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/aws/flatcar-linux/kubernetes/README.md
+++ b/aws/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/flatcar-linux/kubernetes/ami.tf
+++ b/aws/flatcar-linux/kubernetes/ami.tf
@ -1,7 +1,7 @@
 locals {
  # Pick a Flatcar Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id  = data.aws_ami.flatcar.image_id
+  ami_id  = var.arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
  channel = split("-", var.os_image)[1]
 }

@ -25,3 +25,25 @@ data "aws_ami" "flatcar" {
  }
 }

+data "aws_ami" "flatcar-arm64" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
+  }
+}
+
--- a/aws/flatcar-linux/kubernetes/bootstrap.tf
+++ b/aws/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -12,5 +12,6 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }

--- a/aws/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/aws/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -57,9 +57,9 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -70,15 +70,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -87,17 +87,19 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -119,7 +121,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/aws/flatcar-linux/kubernetes/security.tf
+++ b/aws/flatcar-linux/kubernetes/security.tf
@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
  source_security_group_id = aws_security_group.worker.id
 }

@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
  source_security_group_id = aws_security_group.worker.id
 }

--- a/aws/flatcar-linux/kubernetes/ssh.tf
+++ b/aws/flatcar-linux/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/aws/flatcar-linux/kubernetes/variables.tf
+++ b/aws/flatcar-linux/kubernetes/variables.tf
@ -55,19 +55,19 @@ variable "os_image" {
 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -142,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -160,3 +160,19 @@ variable "cluster_domain_suffix" {
  default     = "cluster.local"
 }

+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
+
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
--- a/aws/flatcar-linux/kubernetes/versions.tf
+++ b/aws/flatcar-linux/kubernetes/versions.tf
@ -1,15 +1,15 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/aws/flatcar-linux/kubernetes/workers.tf
+++ b/aws/flatcar-linux/kubernetes/workers.tf
@ -9,6 +9,7 @@ module "workers" {
  worker_count    = var.worker_count
  instance_type   = var.worker_type
  os_image        = var.os_image
+  arch            = var.arch
  disk_size       = var.disk_size
  spot_price      = var.worker_price
  target_groups   = var.worker_target_groups
--- a/aws/flatcar-linux/kubernetes/workers/ami.tf
+++ b/aws/flatcar-linux/kubernetes/workers/ami.tf
@ -1,7 +1,7 @@
 locals {
  # Pick a Flatcar Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id  = data.aws_ami.flatcar.image_id
+  ami_id  = var.arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
  channel = split("-", var.os_image)[1]
 }

@ -25,3 +25,24 @@ data "aws_ami" "flatcar" {
  }
 }

+data "aws_ami" "flatcar-arm64" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
+  }
+}
--- a/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -29,9 +29,9 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -45,15 +45,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -62,20 +62,25 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
          %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -91,7 +96,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/aws/flatcar-linux/kubernetes/workers/variables.tf
+++ b/aws/flatcar-linux/kubernetes/workers/variables.tf
@ -48,13 +48,13 @@ variable "os_image" {
 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
@ -113,3 +113,22 @@ variable "node_labels" {
  description = "List of initial node labels"
  default     = []
 }
+
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
--- a/aws/flatcar-linux/kubernetes/workers/versions.tf
+++ b/aws/flatcar-linux/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/aws/flatcar-linux/kubernetes/workers/workers.tf
+++ b/aws/flatcar-linux/kubernetes/workers/workers.tf
@ -86,6 +86,7 @@ data "template_file" "worker-config" {
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
    node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
  }
 }

--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -18,8 +18,6 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
+  daemonset_tolerations = var.daemonset_tolerations
 }

--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -29,8 +29,10 @@ systemd:
        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -51,8 +53,8 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -63,14 +65,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -82,17 +84,18 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -118,7 +121,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.23.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -213,6 +216,26 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
--- a/azure/fedora-coreos/kubernetes/lb.tf
+++ b/azure/fedora-coreos/kubernetes/lb.tf
@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
@ -112,16 +112,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {

 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "controller"
  loadbalancer_id = azurerm_lb.cluster.id
 }

 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "worker"
  loadbalancer_id = azurerm_lb.cluster.id
 }
--- a/azure/fedora-coreos/kubernetes/security.tf
+++ b/azure/fedora-coreos/kubernetes/security.tf
@ -95,7 +95,7 @@ resource "azurerm_network_security_rule" "controller-kube-metrics" {
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
-  destination_port_range      = "10251-10252"
+  destination_port_range      = "10257-10259"
  source_address_prefix       = azurerm_subnet.worker.address_prefix
  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
--- a/azure/fedora-coreos/kubernetes/ssh.tf
+++ b/azure/fedora-coreos/kubernetes/ssh.tf
@ -25,7 +25,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/azure/fedora-coreos/kubernetes/variables.tf
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@ -54,7 +54,7 @@ variable "os_image" {
 variable "disk_size" {
  type        = number
  description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }

 variable "worker_priority" {
@ -65,13 +65,13 @@ variable "worker_priority" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -117,8 +117,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -135,3 +135,8 @@ variable "cluster_domain_suffix" {
  default     = "cluster.local"
 }

+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -1,15 +1,15 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -1,10 +1,12 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -24,8 +26,8 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -36,14 +38,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -55,20 +57,24 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
          %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -83,7 +89,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -122,10 +128,29 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ${ssh_authorized_key}

-
--- a/azure/fedora-coreos/kubernetes/workers/variables.tf
+++ b/azure/fedora-coreos/kubernetes/workers/variables.tf
@ -57,7 +57,7 @@ variable "priority" {

 variable "snippets" {
  type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
  default     = []
 }

@ -88,6 +88,12 @@ variable "node_labels" {
  default     = []
 }

+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported

 variable "cluster_domain_suffix" {
--- a/azure/fedora-coreos/kubernetes/workers/versions.tf
+++ b/azure/fedora-coreos/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/azure/fedora-coreos/kubernetes/workers/workers.tf
+++ b/azure/fedora-coreos/kubernetes/workers/workers.tf
@ -87,6 +87,7 @@ data "template_file" "worker-config" {
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
    node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
  }
 }

--- a/azure/flatcar-linux/kubernetes/README.md
+++ b/azure/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/flatcar-linux/kubernetes/bootstrap.tf
+++ b/azure/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -18,5 +18,6 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }

--- a/azure/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/azure/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -55,8 +55,8 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -67,15 +67,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -84,16 +84,18 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -115,7 +117,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/azure/flatcar-linux/kubernetes/lb.tf
+++ b/azure/flatcar-linux/kubernetes/lb.tf
@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
@ -112,16 +112,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {

 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "controller"
  loadbalancer_id = azurerm_lb.cluster.id
 }

 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "worker"
  loadbalancer_id = azurerm_lb.cluster.id
 }
--- a/azure/flatcar-linux/kubernetes/security.tf
+++ b/azure/flatcar-linux/kubernetes/security.tf
@ -95,7 +95,7 @@ resource "azurerm_network_security_rule" "controller-kube-metrics" {
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
-  destination_port_range      = "10251-10252"
+  destination_port_range      = "10257-10259"
  source_address_prefix       = azurerm_subnet.worker.address_prefix
  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
--- a/azure/flatcar-linux/kubernetes/ssh.tf
+++ b/azure/flatcar-linux/kubernetes/ssh.tf
@ -25,7 +25,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/azure/flatcar-linux/kubernetes/variables.tf
+++ b/azure/flatcar-linux/kubernetes/variables.tf
@ -60,7 +60,7 @@ variable "os_image" {
 variable "disk_size" {
  type        = number
  description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }

 variable "worker_priority" {
@ -123,8 +123,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -141,3 +141,8 @@ variable "cluster_domain_suffix" {
  default     = "cluster.local"
 }

+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
--- a/azure/flatcar-linux/kubernetes/versions.tf
+++ b/azure/flatcar-linux/kubernetes/versions.tf
@ -1,15 +1,15 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -27,8 +27,8 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -42,15 +42,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -59,19 +59,24 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
          %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -87,7 +92,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/azure/flatcar-linux/kubernetes/workers/variables.tf
+++ b/azure/flatcar-linux/kubernetes/workers/variables.tf
@ -94,6 +94,12 @@ variable "node_labels" {
  default     = []
 }

+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported

 variable "cluster_domain_suffix" {
--- a/azure/flatcar-linux/kubernetes/workers/versions.tf
+++ b/azure/flatcar-linux/kubernetes/workers/versions.tf
@ -1,14 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }
  }
 }
--- a/azure/flatcar-linux/kubernetes/workers/workers.tf
+++ b/azure/flatcar-linux/kubernetes/workers/workers.tf
@ -105,6 +105,7 @@ data "template_file" "worker-config" {
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
    node_labels            = join(",", var.node_labels)
+    node_taints            = join(",", var.node_taints)
  }
 }

--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
@ -13,8 +13,6 @@ module "bootstrap" {
  cluster_domain_suffix           = var.cluster_domain_suffix
  enable_reporting                = var.enable_reporting
  enable_aggregation              = var.enable_aggregation
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }


--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -29,8 +29,10 @@ systemd:
        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -50,8 +52,8 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -62,14 +64,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -81,18 +83,19 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -120,7 +123,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
@ -223,6 +226,26 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -1,10 +1,12 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -23,8 +25,8 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -35,14 +37,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -54,15 +56,15 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in compact(split(",", node_labels)) ~}
          --node-labels=${label} \
@ -72,6 +74,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -121,6 +124,26 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -44,7 +44,7 @@ resource "matchbox_profile" "controllers" {

  kernel = local.kernel
  initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.controller-ignitions.*.rendered[count.index]
 }
@ -78,7 +78,7 @@ resource "matchbox_profile" "workers" {

  kernel = local.kernel
  initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.worker-ignitions.*.rendered[count.index]
 }
--- a/bare-metal/fedora-coreos/kubernetes/ssh.tf
+++ b/bare-metal/fedora-coreos/kubernetes/ssh.tf
@ -28,17 +28,18 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
      "sudo /opt/bootstrap/layout",
    ]
  }
@ -64,12 +65,13 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
    ]
  }
 }
--- a/bare-metal/fedora-coreos/kubernetes/variables.tf
+++ b/bare-metal/fedora-coreos/kubernetes/variables.tf
@ -57,7 +57,7 @@ EOD

 variable "snippets" {
  type        = map(list(string))
-  description = "Map from machine names to lists of Fedora CoreOS Config snippets"
+  description = "Map from machine names to lists of Butane snippets"
  default     = {}
 }

@ -146,8 +146,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/bare-metal/fedora-coreos/kubernetes/versions.tf
+++ b/bare-metal/fedora-coreos/kubernetes/versions.tf
@ -1,19 +1,19 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }

    matchbox = {
      source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
    }
  }
 }
--- a/bare-metal/flatcar-linux/kubernetes/README.md
+++ b/bare-metal/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -63,8 +63,8 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -75,15 +75,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -92,17 +92,19 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -124,7 +126,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
@ -35,8 +35,8 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -50,15 +50,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -67,14 +67,15 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in compact(split(",", node_labels)) ~}
          --node-labels=${label} \
@ -84,6 +85,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
--- a/bare-metal/flatcar-linux/kubernetes/ssh.tf
+++ b/bare-metal/flatcar-linux/kubernetes/ssh.tf
@ -29,17 +29,17 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo /opt/bootstrap/layout",
    ]
  }
@ -66,12 +66,12 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
    ]
  }
 }
--- a/bare-metal/flatcar-linux/kubernetes/variables.tf
+++ b/bare-metal/flatcar-linux/kubernetes/variables.tf
@ -151,8 +151,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/bare-metal/flatcar-linux/kubernetes/versions.tf
+++ b/bare-metal/flatcar-linux/kubernetes/versions.tf
@ -1,19 +1,19 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }

    matchbox = {
      source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
    }
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -17,8 +17,5 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/digital-ocean/fedora-coreos/kubernetes/controllers.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/controllers.tf
@ -41,7 +41,6 @@ resource "digitalocean_droplet" "controllers" {
  size  = var.controller_type

  # network
-  private_networking = true
  vpc_uuid           = digitalocean_vpc.network.id
  # TODO: Only official DigitalOcean images support IPv6
  ipv6 = false
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -29,8 +29,10 @@ systemd:
        LimitNOFILE=40000
        [Install]
        WantedBy=multi-user.target
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -52,9 +54,9 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -65,14 +67,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -84,18 +86,19 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -130,7 +133,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.20.5
+            quay.io/poseidon/kubelet:v1.23.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -220,3 +223,24 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
+
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -1,10 +1,12 @@
 ---
 variant: fcos
-version: 1.1.0
+version: 1.4.0
 systemd:
  units:
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -26,9 +28,9 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/afterburn
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -39,14 +41,14 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
@ -58,18 +60,19 @@ systemd:
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
          --cgroups-per-qos=true \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --enforce-node-allocatable=pods \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -93,7 +96,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -127,3 +130,23 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
--- a/digital-ocean/fedora-coreos/kubernetes/network.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/network.tf
@ -116,7 +116,7 @@ resource "digitalocean_firewall" "controllers" {
  # kube-scheduler metrics, kube-controller-manager metrics
  inbound_rule {
    protocol    = "tcp"
-    port_range  = "10251-10252"
+    port_range  = "10257-10259"
    source_tags = [digitalocean_tag.workers.name]
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/ssh.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/ssh.tf
@ -25,17 +25,18 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
      "sudo /opt/bootstrap/layout",
    ]
  }
@ -54,12 +55,13 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo touch /etc/kubernetes",
    ]
  }
 }
@ -84,4 +86,3 @@ resource "null_resource" "bootstrap" {
    ]
  }
 }
-
--- a/digital-ocean/fedora-coreos/kubernetes/variables.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/variables.tf
@ -48,13 +48,13 @@ variable "os_image" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -94,8 +94,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/digital-ocean/fedora-coreos/kubernetes/versions.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/versions.tf
@ -1,19 +1,19 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.13.0, < 0.15.0"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
-      version = "~> 0.6"
+      version = "~> 0.9"
    }

    digitalocean = {
      source  = "digitalocean/digitalocean"
-      version = "~> 1.20"
+      version = ">= 2.12, < 3.0"
    }
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/workers.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/workers.tf
@ -37,7 +37,6 @@ resource "digitalocean_droplet" "workers" {
  size  = var.worker_type

  # network
-  private_networking = true
  vpc_uuid           = digitalocean_vpc.network.id
  # TODO: Only official DigitalOcean images support IPv6
  ipv6 = false
--- a/digital-ocean/flatcar-linux/kubernetes/README.md
+++ b/digital-ocean/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.20.5 (upstream)
+* Kubernetes v1.23.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=8c2e766d180824416075f4d7a695d6291ef277ab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=f45deec67e2fea4f06b5a3edad628b0fe0e9ec60"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.15
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -65,9 +65,9 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -78,15 +78,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -95,17 +95,19 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -127,7 +129,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
@ -37,9 +37,9 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        EnvironmentFile=/run/metadata/coreos
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -53,15 +53,15 @@ systemd:
          --privileged \
          --pid host \
          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
          -v /etc/kubernetes:/etc/kubernetes:ro \
          -v /etc/machine-id:/etc/machine-id:ro \
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -70,17 +70,19 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --healthz-port=0 \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -96,7 +98,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.20.5
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.2
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/digital-ocean/flatcar-linux/kubernetes/controllers.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/controllers.tf
@ -46,7 +46,6 @@ resource "digitalocean_droplet" "controllers" {
  size  = var.controller_type

  # network
-  private_networking = true
  vpc_uuid           = digitalocean_vpc.network.id
  # TODO: Only official DigitalOcean images support IPv6
  ipv6 = false
--- a/digital-ocean/flatcar-linux/kubernetes/network.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/network.tf
@ -116,7 +116,7 @@ resource "digitalocean_firewall" "controllers" {
  # kube-scheduler metrics, kube-controller-manager metrics
  inbound_rule {
    protocol    = "tcp"
-    port_range  = "10251-10252"
+    port_range  = "10257-10259"
    source_tags = [digitalocean_tag.workers.name]
  }
 }
--- a/digital-ocean/flatcar-linux/kubernetes/ssh.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/ssh.tf
@ -25,17 +25,17 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo /opt/bootstrap/layout",
    ]
  }
@ -54,12 +54,12 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
    ]
  }
 }
--- a/Show More
+++ b/Show More