Update Prometheus and Grafana addons

Fix Azure backend_address_pool_id deprecation warning
* Change to `backend_address_pool_ids` list
2025-08-02 17:51:33 +02:00 · 2021-12-15 08:16:46 -08:00 · 2021-12-14 10:26:08 -08:00 · 2021-12-13 17:44:02 -08:00 · 2021-12-10 11:29:49 -08:00 · 2021-12-10 11:26:05 -08:00
117 changed files with 595 additions and 501 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,132 @@ Notable changes between versions.

 ## Latest

+* Kubernetes [v1.23.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230)
+* Normalize CA cert mounts in static Pods and kube-proxy ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+* Set Kubelet resolver config to `/run/systemd/resolve/resolv.conf` ([#1082](https://github.com/poseidon/typhoon/pull/1082))
+* Update Cilium from v1.10.5 to [v1.11.0](https://github.com/cilium/cilium/releases/tag/v1.11.0) ([#1083](https://github.com/poseidon/typhoon/pull/1083))
+* With Calico, add missing `caliconodestatuses` CRD ([#289](https://github.com/poseidon/terraform-render-bootstrap/pull/289))
+* Change `enable_aggregation` default to true ([#279](https://github.com/poseidon/terraform-render-bootstrap/pull/279))
+* Remove deprecated `--port` from `kube-scheduler` ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+
+### AWS
+
+* Change controller node default `disk_iops` to 3000 ([#1073](https://github.com/poseidon/typhoon/pull/1073))
+
+### Azure
+
+* Fix warning about deprecated `backend_address_pool_id` ([#1086](https://github.com/poseidon/typhoon/pull/1086))
+
+### Fedora CoreOS
+
+* Fix Fedora ARM64 workers to official Fedora CoreOS AMIs ([#1072](https://github.com/poseidon/typhoon/pull/1072))
+  * Should have been changed alongside controller AMIs in ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+  * Old Poseidon built ARM64 AMIs have been deleted
+
+### Addons
+
+* Update nginx-ingress from v1.0.5 to [v1.1.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.0)
+* Update Prometheus from v2.31.1 to [v2.32.0](https://github.com/prometheus/prometheus/releases/tag/v2.32.0)
+* Update kube-state-metrics from v2.2.4 to [v2.3.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.3.0)
+* Update node-exporter from v1.3.0 to [v1.3.1](https://github.com/prometheus/node_exporter/releases/tag/v1.3.1)
+* Update Grafana from v8.2.4 to [v8.3.3](https://github.com/grafana/grafana/releases/tag/v8.3.3)
+
+### Known Issues
+
+* Calico does not yet support Kubernetes v1.23.0, use `flannel` or `cilium` ([calico#5011](https://github.com/projectcalico/calico/issues/5011))
+
+## v1.22.4
+
+* Kubernetes [v1.22.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224)
+* Update CoreDNS from v1.8.4 to [v1.8.6](https://github.com/poseidon/terraform-render-bootstrap/pull/284)
+* Update Calico from v3.20.2 to [v3.21.0](https://github.com/projectcalico/calico/releases/tag/v3.21.0)
+* Update flannel from v0.14.0 to [v0.15.1](https://github.com/flannel-io/flannel/releases/tag/v0.15.1)
+
+### Google
+
+* Allow use of Terraform provider `google` [v4.0+](https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0)
+
+### Flatcar Linux
+
+* Change Kubelet mounts for cgroups v2 ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+* Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+
+### Addons
+
+* Update Prometheus from v2.30.3 to [v2.31.1](https://github.com/prometheus/prometheus/releases/tag/v2.31.1)
+* Update node-exporter from v1.2.2 to [v1.3.0](https://github.com/prometheus/node_exporter/releases/tag/v1.3.0)
+* Update kube-state-metrics from v2.2.3 to [v2.2.4](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.4)
+* Update Grafana from v8.2.1 to [v8.2.4](https://github.com/grafana/grafana/releases/tag/v8.2.4)
+* Update nginx-ingress from v1.0.4 to [v1.0.5](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.5)
+
+## v1.23.3
+
+* Kubernetes [v1.22.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223)
+* Update etcd from v3.5.0 to [v3.5.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.1)
+* Update Cilium from v1.10.4 to [v1.10.5](https://github.com/cilium/cilium/releases/tag/v1.10.5)
+* Update Calico from v3.20.1 to [v3.20.2](https://github.com/projectcalico/calico/releases/tag/v3.20.2)
+  * Use Calico's iptables legacy vs nft auto-detection
+* Update flannel from v0.13.0 to v0.14.0
+
+### Bare-Metal
+
+* Require Terraform provider `poseidon/matchbox` v0.5+ ([#1048](https://github.com/poseidon/typhoon/pull/1048))
+
+### Addons
+
+* Update nginx-ingress from v1.0.0 to [v1.0.4](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4)
+* Update Prometheus from v2.29.2 to [v2.30.3](https://github.com/prometheus/prometheus/releases/tag/v2.30.3)
+* Update kube-state-metrics from v2.2.0 to [v2.2.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.3)
+* Update Grafana from v8.1.2 to [v8.2.1](https://github.com/grafana/grafana/releases/tag/v8.2.1)
+
+## v1.22.2
+
+* Kubernetes [v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222)
+* Update Cilium from v1.10.3 to [v1.10.4](https://github.com/cilium/cilium/releases/tag/v1.10.4)
+* Update Calico from v3.20.0 to [v3.20.1](https://github.com/projectcalico/calico/releases/tag/v3.20.1)
+* Fix access to ClusterIP services with Cilium ([#276](https://github.com/poseidon/terraform-render-bootstrap/pull/276))
+
+### Fedora CoreOS
+
+* Use Fedora CoreOS ARM64 AMIs ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+
+### Addons
+
+* Update Prometheus from v2.29.1 to [v2.29.2](https://github.com/prometheus/prometheus/releases/tag/v2.29.2)
+* Update kube-state-metrics from v2.1.1 to [v2.2.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.0)
+
+## v1.22.1
+
+* Kubernetes [v1.22.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221)
+* Update Calico from v3.19.1 to [v3.20.0](https://github.com/projectcalico/calico/releases/tag/v3.20.0)
+
+### Addons
+
+* Update nginx-ingress from v1.0.0-beta.1 to [v1.0.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0)
+* Update Prometheus from v2.28.1 to [v2.29.1](https://github.com/prometheus/prometheus/releases/tag/v2.29.1)
+* Update Grafana from v8.1.1 to [v8.1.2](https://github.com/grafana/grafana/releases/tag/v8.1.2)
+
+## v1.22.0
+
+* Kubernetes [v1.22.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1220)
+* Update etcd from v3.4.16 to [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0)
+* Switch `kube-controller-manager` and `kube-scheduler` to use secure port only
+  * Update Prometheus config to discover endpoints and use a bearer token to scrape
+
+### Fedora CoreOS
+
+* Add Cilium cgroups v2 support on Fedora CoreOS
+* Update Butane Config version from v1.2.0 to v1.4.0
+  * Rename Fedora CoreOS Config to Butane Config
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.4.0
+
+### Addons
+
+* Update nginx-ingress from v0.47.0 to [v1.0.0-beta.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0-beta.1)
+* Update node-exporter from v1.2.0 to [v1.2.2](https://github.com/prometheus/node_exporter/releases/tag/v1.2.2)
+* Update kube-state-metrics from v2.1.0 to [v2.1.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.1)
+* Update Grafana from v8.0.6 to [v8.1.1](https://github.com/grafana/grafana/releases/tag/v8.1.1)
+
 ## v1.21.3

 * Kubernetes [v1.21.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1213)
@ -20,6 +146,10 @@ Notable changes between versions.
 * Update node-exporter from v1.1.2 to [v1.2.0](https://github.com/prometheus/node_exporter/releases/tag/v1.2.0)
 * Update Grafana from v8.0.3 to [v8.0.6](https://github.com/grafana/grafana/releases/tag/v8.0.6)

+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
 ## v1.21.2

 * Kubernetes [v1.21.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1212)
@ -53,7 +183,7 @@ Notable changes between versions.

 ### Known Issues

-* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881))
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)

 ## v1.21.1

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@ -58,7 +58,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -67,7 +67,7 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count = 2
@ -97,9 +97,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.3
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.3
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -24,7 +24,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:8.0.6
+          image: docker.io/grafana/grafana:8.3.3
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v0.47.0
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/prometheus/config.yaml
+++ b/addons/prometheus/config.yaml
@ -72,6 +72,48 @@ data:
        regex: apiserver_request_duration_seconds_count;.+
        action: drop

+    # Scrape config for kube-controller-manager endpoints.
+    #
+    # kube-controller-manager service endpoints can be discovered by using the
+    # `endpoints` role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-controller-manager and the `https` port.
+    - job_name: 'kube-controller-manager'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-controller-manager;metrics
+      - replacement: kube-controller-manager
+        action: replace
+        target_label: job
+
+    # Scrape config for kube-scheduler endpoints.
+    #
+    # kube-scheduler service endpoints can be discovered by using the `endpoints`
+    # role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-scheduler and the `https` port.
+    - job_name: 'kube-scheduler'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-scheduler;metrics
+      - replacement: kube-scheduler
+        action: replace
+        target_label: job
+
    # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
    # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
    - job_name: 'kubelet'
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -21,7 +21,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.28.1
+          image: quay.io/prometheus/prometheus:v2.32.0
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/discovery/kube-controller-manager.yaml
+++ b/addons/prometheus/discovery/kube-controller-manager.yaml
@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
  name: kube-controller-manager
  namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
  clusterIP: None
@ -14,5 +12,5 @@ spec:
  ports:
    - name: metrics
      protocol: TCP
-      port: 10252
-      targetPort: 10252
+      port: 10257
+      targetPort: 10257
--- a/addons/prometheus/discovery/kube-scheduler.yaml
+++ b/addons/prometheus/discovery/kube-scheduler.yaml
@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
  name: kube-scheduler
  namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
  clusterIP: None
@ -14,5 +12,5 @@ spec:
  ports:
    - name: metrics
      protocol: TCP
-      port: 10251
-      targetPort: 10251
+      port: 10259
+      targetPort: 10259
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -25,7 +25,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.0
+        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -28,7 +28,7 @@ spec:
      hostPID: true
      containers:
      - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.2.0
+        image: quay.io/prometheus/node-exporter:v1.3.1
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
  }
 }

-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
  count = var.arch == "arm64" ? 1 : 0

  most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]

  filter {
    name   = "architecture"
@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
  }

  filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
-
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -13,7 +13,5 @@ module "bootstrap" {
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
  daemonset_tolerations = var.daemonset_tolerations
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/aws/fedora-coreos/kubernetes/controllers.tf
+++ b/aws/fedora-coreos/kubernetes/controllers.tf
@ -62,7 +62,6 @@ data "template_file" "controller-configs" {

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
-    etcd_arch   = var.arch == "arm64" ? "-arm64" : ""
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16${etcd_arch}
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -54,7 +54,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -97,6 +97,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -122,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.21.3
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -218,6 +219,7 @@ storage:
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
          ETCD_UNSUPPORTED_ARCH=arm64
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/security.tf
+++ b/aws/fedora-coreos/kubernetes/security.tf
@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
  source_security_group_id = aws_security_group.worker.id
 }

@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
  source_security_group_id = aws_security_group.worker.id
 }

--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -66,8 +66,8 @@ variable "disk_type" {

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -84,13 +84,13 @@ variable "worker_target_groups" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -142,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/aws/fedora-coreos/kubernetes/versions.tf
+++ b/aws/fedora-coreos/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
  }
 }

-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
  count = var.arch == "arm64" ? 1 : 0

  most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]

  filter {
    name   = "architecture"
@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
  }

  filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
-
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: docker.service
@ -27,7 +27,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -76,6 +76,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -90,7 +91,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -129,6 +130,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/workers/variables.tf
+++ b/aws/fedora-coreos/kubernetes/workers/variables.tf
@ -77,7 +77,7 @@ variable "target_groups" {

 variable "snippets" {
  type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
  default     = []
 }

--- a/aws/fedora-coreos/kubernetes/workers/versions.tf
+++ b/aws/fedora-coreos/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/aws/flatcar-linux/kubernetes/README.md
+++ b/aws/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/flatcar-linux/kubernetes/bootstrap.tf
+++ b/aws/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/aws/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -57,7 +57,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -76,8 +76,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -88,6 +87,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -98,6 +98,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -119,7 +120,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/aws/flatcar-linux/kubernetes/security.tf
+++ b/aws/flatcar-linux/kubernetes/security.tf
@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
  source_security_group_id = aws_security_group.worker.id
 }

@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
  source_security_group_id = aws_security_group.worker.id
 }

--- a/aws/flatcar-linux/kubernetes/variables.tf
+++ b/aws/flatcar-linux/kubernetes/variables.tf
@ -66,8 +66,8 @@ variable "disk_type" {

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -142,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/aws/flatcar-linux/kubernetes/versions.tf
+++ b/aws/flatcar-linux/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -29,7 +29,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -51,8 +51,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -63,6 +62,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -79,6 +79,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -94,7 +95,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/aws/flatcar-linux/kubernetes/workers/versions.tf
+++ b/aws/flatcar-linux/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -19,8 +19,5 @@ module "bootstrap" {
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
  daemonset_tolerations = var.daemonset_tolerations
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -51,7 +51,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -92,6 +92,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -117,7 +118,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.21.3
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -212,6 +213,7 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/azure/fedora-coreos/kubernetes/lb.tf
+++ b/azure/fedora-coreos/kubernetes/lb.tf
@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
--- a/azure/fedora-coreos/kubernetes/security.tf
+++ b/azure/fedora-coreos/kubernetes/security.tf
@ -95,7 +95,7 @@ resource "azurerm_network_security_rule" "controller-kube-metrics" {
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
-  destination_port_range      = "10251-10252"
+  destination_port_range      = "10257-10259"
  source_address_prefix       = azurerm_subnet.worker.address_prefix
  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
--- a/azure/fedora-coreos/kubernetes/variables.tf
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@ -65,13 +65,13 @@ variable "worker_priority" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -117,8 +117,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: docker.service
@ -24,7 +24,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -71,6 +71,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -85,7 +86,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -124,6 +125,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/azure/fedora-coreos/kubernetes/workers/variables.tf
+++ b/azure/fedora-coreos/kubernetes/workers/variables.tf
@ -57,7 +57,7 @@ variable "priority" {

 variable "snippets" {
  type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
  default     = []
 }

--- a/azure/fedora-coreos/kubernetes/workers/versions.tf
+++ b/azure/fedora-coreos/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/azure/flatcar-linux/kubernetes/README.md
+++ b/azure/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/flatcar-linux/kubernetes/bootstrap.tf
+++ b/azure/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/azure/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -55,7 +55,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -73,8 +73,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -85,6 +84,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -94,6 +94,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -115,7 +116,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/azure/flatcar-linux/kubernetes/lb.tf
+++ b/azure/flatcar-linux/kubernetes/lb.tf
@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
--- a/azure/flatcar-linux/kubernetes/security.tf
+++ b/azure/flatcar-linux/kubernetes/security.tf
@ -95,7 +95,7 @@ resource "azurerm_network_security_rule" "controller-kube-metrics" {
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
-  destination_port_range      = "10251-10252"
+  destination_port_range      = "10257-10259"
  source_address_prefix       = azurerm_subnet.worker.address_prefix
  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }
--- a/azure/flatcar-linux/kubernetes/variables.tf
+++ b/azure/flatcar-linux/kubernetes/variables.tf
@ -123,8 +123,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/azure/flatcar-linux/kubernetes/versions.tf
+++ b/azure/flatcar-linux/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -48,8 +48,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -60,6 +59,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -75,6 +75,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -90,7 +91,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/azure/flatcar-linux/kubernetes/workers/versions.tf
+++ b/azure/flatcar-linux/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
@ -13,8 +13,6 @@ module "bootstrap" {
  cluster_domain_suffix           = var.cluster_domain_suffix
  enable_reporting                = var.enable_reporting
  enable_aggregation              = var.enable_aggregation
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }


--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -50,7 +50,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -92,6 +92,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -119,7 +120,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
@ -222,6 +223,7 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: docker.service
@ -23,7 +23,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -71,6 +71,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -120,6 +121,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -44,7 +44,7 @@ resource "matchbox_profile" "controllers" {

  kernel = local.kernel
  initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.controller-ignitions.*.rendered[count.index]
 }
@ -78,7 +78,7 @@ resource "matchbox_profile" "workers" {

  kernel = local.kernel
  initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.worker-ignitions.*.rendered[count.index]
 }
--- a/bare-metal/fedora-coreos/kubernetes/variables.tf
+++ b/bare-metal/fedora-coreos/kubernetes/variables.tf
@ -57,7 +57,7 @@ EOD

 variable "snippets" {
  type        = map(list(string))
-  description = "Map from machine names to lists of Fedora CoreOS Config snippets"
+  description = "Map from machine names to lists of Butane snippets"
  default     = {}
 }

@ -146,8 +146,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/bare-metal/fedora-coreos/kubernetes/versions.tf
+++ b/bare-metal/fedora-coreos/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
@ -13,7 +13,7 @@ terraform {

    matchbox = {
      source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
    }
  }
 }
--- a/bare-metal/flatcar-linux/kubernetes/README.md
+++ b/bare-metal/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -63,7 +63,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -81,8 +81,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -93,6 +92,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -103,6 +103,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -124,7 +125,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
@ -35,7 +35,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -56,8 +56,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -68,6 +67,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -84,6 +84,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
--- a/bare-metal/flatcar-linux/kubernetes/variables.tf
+++ b/bare-metal/flatcar-linux/kubernetes/variables.tf
@ -151,8 +151,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/bare-metal/flatcar-linux/kubernetes/versions.tf
+++ b/bare-metal/flatcar-linux/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
@ -13,7 +13,7 @@ terraform {

    matchbox = {
      source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
    }
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -17,8 +17,5 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -52,7 +52,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -95,6 +95,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -129,7 +130,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.21.3
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -219,3 +220,5 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 systemd:
  units:
    - name: docker.service
@ -26,7 +26,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -69,6 +69,7 @@ systemd:
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -92,7 +93,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -126,3 +127,4 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
--- a/digital-ocean/fedora-coreos/kubernetes/network.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/network.tf
@ -116,7 +116,7 @@ resource "digitalocean_firewall" "controllers" {
  # kube-scheduler metrics, kube-controller-manager metrics
  inbound_rule {
    protocol    = "tcp"
-    port_range  = "10251-10252"
+    port_range  = "10257-10259"
    source_tags = [digitalocean_tag.workers.name]
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/variables.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/variables.tf
@ -48,13 +48,13 @@ variable "os_image" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -94,8 +94,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/digital-ocean/fedora-coreos/kubernetes/versions.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/digital-ocean/flatcar-linux/kubernetes/README.md
+++ b/digital-ocean/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=5746f9c221fb779def042c81ea827fed1b844f1d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.4.16
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -65,7 +65,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -84,8 +84,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -96,6 +95,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -106,6 +106,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -127,7 +128,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
@ -37,7 +37,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -59,8 +59,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -71,6 +70,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -81,6 +81,7 @@ systemd:
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -96,7 +97,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.21.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/digital-ocean/flatcar-linux/kubernetes/network.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/network.tf
@ -116,7 +116,7 @@ resource "digitalocean_firewall" "controllers" {
  # kube-scheduler metrics, kube-controller-manager metrics
  inbound_rule {
    protocol    = "tcp"
-    port_range  = "10251-10252"
+    port_range  = "10257-10259"
    source_tags = [digitalocean_tag.workers.name]
  }
 }
--- a/digital-ocean/flatcar-linux/kubernetes/variables.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/variables.tf
@ -94,8 +94,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/digital-ocean/flatcar-linux/kubernetes/versions.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/docs/advanced/arm64.md
+++ b/docs/advanced/arm64.md
@ -8,20 +8,13 @@ Typhoon has experimental support for ARM64 with Fedora CoreOS on AWS. Full clust
 !!! note
    Currently, CNI networking must be set to flannel or Cilium.

-## AMIs
-
-In lieu of official Fedora CoreOS ARM64 AMIs, Poseidon publishes experimental ARM64 AMIs to a few regions (us-east-1, us-east-2, us-west-1). These AMIs may be **removed** at any time and will be replaced when Fedora CoreOS publishes equivalents.
-
-!!! note
-    AMIs are only published to a few regions, and AWS availability of ARM instance types varies.
-
 ## Cluster

 Create a cluster with ARM64 controller and worker nodes. Container workloads must be `arm64` compatible and use `arm64` container images.

 ```tf
 module "gravitas" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"

  # AWS
  cluster_name = "gravitas"
@ -29,7 +22,7 @@ module "gravitas" {
  dns_zone_id  = "Z3PAABBCFAKEC0"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  arch         = "arm64"
@ -47,9 +40,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes.
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-12-178   Ready    <none>   101s   v1.21.3   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-18-93    Ready    <none>   102s   v1.21.3   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-90-10    Ready    <none>   104s   v1.21.3   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-12-178   Ready    <none>   101s   v1.23.0   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-18-93    Ready    <none>   102s   v1.23.0   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-90-10    Ready    <none>   104s   v1.23.0   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
 ```

 ## Hybrid
@ -60,7 +53,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo

    ```tf
    module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"

      # AWS
      cluster_name = "gravitas"
@ -68,7 +61,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
      dns_zone_id  = "Z3PAABBCFAKEC0"

      # configuration
-      ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

      # optional
      networking   = "cilium"
@ -83,7 +76,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo

    ```tf
    module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # AWS
      vpc_id          = module.gravitas.vpc_id
@ -108,9 +101,9 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 ```
 $ kubectl get nodes -o wide
 NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION             CONTAINER-RUNTIME
-ip-10-0-1-81    Ready    <none>   4m28s   v1.21.3   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-17-86   Ready    <none>   4m28s   v1.21.3   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
-ip-10-0-21-45   Ready    <none>   4m28s   v1.21.3   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-40-36   Ready    <none>   4m22s   v1.21.3   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-1-81    Ready    <none>   4m28s   v1.23.0   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-17-86   Ready    <none>   4m28s   v1.23.0   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
+ip-10-0-21-45   Ready    <none>   4m28s   v1.23.0   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-40-36   Ready    <none>   4m22s   v1.23.0   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
 ```

--- a/docs/advanced/customization.md
+++ b/docs/advanced/customization.md
@ -12,9 +12,9 @@ Clusters are kept to a minimal Kubernetes control plane by offering components l

 ## Hosts

-Typhoon uses the [Ignition](https://github.com/coreos/ignition) system of Fedora CoreOS and Flatcar Linux to immutably declare a system via first-boot disk provisioning. Fedora CoreOS uses a [Fedora CoreOS Config](https://docs.fedoraproject.org/en-US/fedora-coreos/fcct-config/) (FCC) and Flatcar Linux uses a [Container Linux Config](https://github.com/coreos/container-linux-config-transpiler/blob/master/doc/examples.md) (CLC). These define disk partitions, filesystems, systemd units, dropins, config files, mount units, raid arrays, and users.
+Typhoon uses the [Ignition](https://github.com/coreos/ignition) system of Fedora CoreOS and Flatcar Linux to immutably declare a system via first-boot disk provisioning. Fedora CoreOS uses a [Butane Config](https://coreos.github.io/butane/specs/) and Flatcar Linux uses a [Container Linux Config](https://github.com/coreos/container-linux-config-transpiler/blob/master/doc/examples.md) (CLC). These define disk partitions, filesystems, systemd units, dropins, config files, mount units, raid arrays, and users.

-Controller and worker instances form a minimal and secure Kubernetes cluster on each platform. Typhoon provides the **snippets** feature to accept Fedora CoreOS Configs or Container Linux Configs to validate and additively merge into instance declarations. This allows advanced host customization and experimentation.
+Controller and worker instances form a minimal and secure Kubernetes cluster on each platform. Typhoon provides the **snippets** feature to accept Butane or Container Linux Configs to validate and additively merge into instance declarations. This allows advanced host customization and experimentation.

 !!! note
    Snippets cannot be used to modify an already existing instance, the antithesis of immutable provisioning. Ignition fully declares a system on first boot only.
@ -30,14 +30,14 @@ Controller and worker instances form a minimal and secure Kubernetes cluster on
 !!! note
    Fedora CoreOS snippets require `terraform-provider-ct` v0.5+

-Define a Fedora CoreOS Config (FCC) ([docs](https://docs.fedoraproject.org/en-US/fedora-coreos/fcct-config/), [config](https://github.com/coreos/fcct/blob/master/docs/configuration-v1_0.md), [examples](https://github.com/coreos/fcct/blob/master/docs/examples.md)) in version control near your Terraform workspace directory (e.g. perhaps in a `snippets` subdirectory). You may organize snippets into multiple files, if desired.
+Define a Butane Config ([docs](https://coreos.github.io/butane/specs/), [config](https://github.com/coreos/butane/blob/main/docs/config-fcos-v1_4.md)) in version control near your Terraform workspace directory (e.g. perhaps in a `snippets` subdirectory). You may organize snippets into multiple files, if desired.

 For example, ensure an `/opt/hello` file is created with permissions 0644.

 ```yaml
 # custom-files
 variant: fcos
-version: 1.2.0
+version: 1.4.0
 storage:
  files:
    - path: /opt/hello
@ -185,7 +185,7 @@ To set an alternative etcd image or Kubelet image, use a snippet to set a system
    ```yaml
    # kubelet-image-override.yaml
    variant: fcos           <- remove for Flatcar Linux
-    version: 1.2.0          <- remove for Flatcar Linux
+    version: 1.4.0          <- remove for Flatcar Linux
    systemd:
      units:
        - name: kubelet.service
@ -201,7 +201,7 @@ To set an alternative etcd image or Kubelet image, use a snippet to set a system
    ```yaml
    # etcd-image-override.yaml
    variant: fcos           <- remove for Flatcar Linux
-    version: 1.2.0          <- remove for Flatcar Linux
+    version: 1.4.0          <- remove for Flatcar Linux
    systemd:
      units:
        - name: etcd-member.service
--- a/docs/advanced/nodes.md
+++ b/docs/advanced/nodes.md
@ -36,7 +36,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to

    ```tf
    module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

      # Google Cloud
      cluster_name  = "yavin"
@ -57,7 +57,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to

    ```tf
    module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      cluster_name = "yavin"
@ -89,7 +89,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and

    ```tf
    module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

      # Google Cloud
      cluster_name  = "yavin"
@ -110,7 +110,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and

    ```tf
    module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      cluster_name = "yavin"
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).

    ```tf
    module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # AWS
      vpc_id          = module.tempest.vpc_id
@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).

    ```tf
    module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.23.0"

      # AWS
      vpc_id          = module.tempest.vpc_id
@ -82,7 +82,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | subnet_ids | Must be set to `subnet_ids` output by cluster | module.cluster.subnet_ids |
 | security_groups | Must be set to `worker_security_groups` output by cluster | module.cluster.worker_security_groups |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |

 #### Optional

@ -111,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste

    ```tf
    module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Azure
      region                  = module.ramius.region
@ -137,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste

    ```tf
    module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.23.0"

      # Azure
      region                  = module.ramius.region
@ -182,7 +182,7 @@ The Azure internal `workers` module supports a number of [variables](https://git
 | security_group_id | Must be set to `security_group_id` output by cluster | module.cluster.security_group_id |
 | backend_address_pool_id | Must be set to `backend_address_pool_id` output by cluster | module.cluster.backend_address_pool_id |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |

 #### Optional

@ -207,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c

    ```tf
    module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      region       = "europe-west2"
@ -231,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c

    ```tf
    module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.21.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      region       = "europe-west2"
@ -262,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.21.3
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.21.3
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.21.3
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.21.3
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.21.3
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.23.0
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.23.0
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.23.0
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.23.0
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.23.0
 ```

 ### Variables
@ -283,7 +283,7 @@ The Google Cloud internal `workers` module supports a number of [variables](http
 | network | Must be set to `network_name` output by cluster | module.cluster.network_name |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
 | os_image | Container Linux image for compute instances | "uploaded-flatcar-image" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |

 Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`.

--- a/docs/architecture/operating-systems.md
+++ b/docs/architecture/operating-systems.md
@ -15,24 +15,26 @@ Together, they diversify Typhoon to support a range of container technologies.
 ## Host Properties

 | Property          | Flatcar Linux | Fedora CoreOS |
-|-------------------|---------------------------------|---------------|
-| Kernel            | ~5.4.x | ~5.8.x |
-| systemd           | 245 | 245 |
+|-------------------|---------------|---------------|
+| Kernel            | ~5.10.x       | ~5.14.x       |
+| systemd           | 247           | 249           |
+| Username          | core          | core          |
 | Ignition system   | Ignition v2.x spec | Ignition v3.x spec |
-| Container Engine  | docker 19.3.12  | docker 19.03.11 |
 | storage driver    | overlay2 (extfs)  | overlay2 (xfs) |
-| logging driver    | json-file | journald |
-| cgroup driver     | cgroupfs (except Flatcar edge) | systemd  |
-| Networking        | systemd-networkd | NetworkManager |
-| Username          | core      | core |
+| logging driver    | json-file     | journald      |
+| cgroup driver     | systemd       | systemd       |
+| cgroup version    | v2            | v2            |
+| Networking        | systemd-networkd | NetworkManager   |
+| Resolver          | systemd-resolved | systemd-resolved |

 ## Kubernetes Properties

 | Property          | Flatcar Linux | Fedora CoreOS |
-|-------------------|-----------------|---------------|
+|-------------------|---------------|---------------|
 | single-master     | all platforms | all platforms |
 | multi-master      | all platforms | all platforms |
 | control plane     | static pods   | static pods   |
+| Container Runtime | docker 20.10  | docker 20.10  |
 | kubelet image     | kubelet [image](https://github.com/poseidon/kubelet) with upstream binary | kubelet [image](https://github.com/poseidon/kubelet) with upstream binary |
 | control plane images | upstream images | upstream images |
 | on-host etcd      | docker    | podman |
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -51,11 +51,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    aws = {
      source = "hashicorp/aws"
-      version = "3.48.0"
+      version = "3.67.0"
    }
  }
 }
@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"

  # AWS
  cluster_name = "tempest"
@ -80,7 +80,7 @@ module "tempest" {
  dns_zone_id  = "Z3PAABBCFAKEC0"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count = 2
@ -95,7 +95,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.

 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```

@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.21.3
-ip-10-0-26-65  Ready   <none>   10m  v1.21.3
-ip-10-0-41-21  Ready   <none>   10m  v1.21.3
+ip-10-0-3-155  Ready   <none>   10m  v1.23.0
+ip-10-0-26-65  Ready   <none>   10m  v1.23.0
+ip-10-0-41-21  Ready   <none>   10m  v1.23.0
 ```

 List the pods.
@ -183,7 +183,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/aws/fed
 | cluster_name | Unique cluster name (prepended to dns_zone) | "tempest" |
 | dns_zone | AWS Route53 DNS zone | "aws.example.com" |
 | dns_zone_id | AWS Route53 DNS zone id | "Z3PAABBCFAKEC0" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |

 #### DNS Zone

@ -216,8 +216,8 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | worker_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0 | 0.10 |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
+| controller_snippets | Controller Butane snippets | [] | [examples](/advanced/customization/) |
+| worker_snippets | Worker Butane snippets | [] | [examples](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
 | host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |
--- a/docs/fedora-coreos/azure.md
+++ b/docs/fedora-coreos/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -48,11 +48,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    azurerm = {
      source = "hashicorp/azurerm"
-      version = "2.68.0"
+      version = "2.88.1"
    }
  }
 }
@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.23.0"

  # Azure
  cluster_name   = "ramius"
@ -96,7 +96,7 @@ module "ramius" {

  # configuration
  os_image           = "/subscriptions/some/path/Microsoft.Compute/images/fedora-coreos-31.20200323.3.2"
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count    = 2
@ -111,7 +111,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.

 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```

@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.21.3
-ramius-worker-000001  Ready   <none>  25m  v1.21.3
-ramius-worker-000002  Ready   <none>  24m  v1.21.3
+ramius-controller-0   Ready   <none>  24m  v1.23.0
+ramius-worker-000001  Ready   <none>  25m  v1.23.0
+ramius-worker-000002  Ready   <none>  24m  v1.23.0
 ```

 List the pods.
@ -201,7 +201,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/f
 | dns_zone | Azure DNS zone | "azure.example.com" |
 | dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
 | os_image | Fedora CoreOS image for instances | "/subscriptions/..../custom-image" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |

 !!! tip
    Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
@ -243,8 +243,8 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | worker_type | Machine type for workers | "Standard_DS1_v2" | See below |
 | disk_size | Size of the disk in GB | 30 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/#usage) |
+| controller_snippets | Controller Butane snippets | [] | [example](/advanced/customization/#usage) |
+| worker_snippets | Worker Butane snippets | [] | [example](/advanced/customization/#usage) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.21.3 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -138,11 +138,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    matchbox = {
      source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
    }
  }
 }
@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.23.0"

  # bare-metal
  cluster_name            = "mercury"
@ -164,7 +164,7 @@ module "mercury" {

  # configuration
  k8s_domain_name    = "node1.example.com"
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # machines
  controllers = [{
@ -194,7 +194,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.

 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```

@ -283,9 +283,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.21.3
-node2.example.com   Ready   <none>  10m  v1.21.3
-node3.example.com   Ready   <none>  10m  v1.21.3
+node1.example.com   Ready   <none>  10m  v1.23.0
+node2.example.com   Ready   <none>  10m  v1.23.0
+node3.example.com   Ready   <none>  10m  v1.23.0
 ```

 List the pods.
@ -323,7 +323,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | os_stream | Fedora CoreOS release stream | "stable" |
 | os_version | Fedora CoreOS version to PXE and install | "32.20201104.3.0" |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3Nz..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3Nz..." |
 | controllers | List of controller machine detail objects (unique name, identifying MAC address, FQDN) | `[{name="node1", mac="52:54:00:a1:9c:ae", domain="node1.example.com"}]` |
 | workers | List of worker machine detail objects (unique name, identifying MAC address, FQDN) | `[{name="node2", mac="52:54:00:b2:2f:86", domain="node2.example.com"}, {name="node3", mac="52:54:00:c3:61:77", domain="node3.example.com"}]` |

@ -335,7 +335,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | install_disk | Disk device where Fedora CoreOS should be installed | "sda" (not "/dev/sda" like Container Linux) | "sdb" |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | network_mtu | CNI interface MTU (calico-only) | 1480 | - |
-| snippets | Map from machine names to lists of Fedora CoreOS Config snippets | {} | [examples](/advanced/customization/) |
+| snippets | Map from machine names to lists of Butane snippets | {} | [examples](/advanced/customization/) |
 | network_ip_autodetection_method | Method to detect host IPv4 address (calico-only) | "first-found" | "can-reach=10.0.0.1" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@ -1,6 +1,6 @@
 # DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -51,7 +51,7 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    digitalocean = {
      source = "digitalocean/digitalocean"
@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.23.0"

  # Digital Ocean
  cluster_name = "nemo"
@ -104,7 +104,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.

 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```

@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.21.3
-10.132.115.81      Ready   <none>  10m  v1.21.3
-10.132.124.107     Ready   <none>  10m  v1.21.3
+10.132.110.130     Ready   <none>  10m  v1.23.0
+10.132.115.81      Ready   <none>  10m  v1.23.0
+10.132.124.107     Ready   <none>  10m  v1.23.0
 ```

 List the pods.
@ -214,7 +214,7 @@ resource "digitalocean_domain" "zone-for-clusters" {

 #### SSH Fingerprints

-DigitalOcean droplets are created with your SSH public key "fingerprint" (i.e. MD5 hash) to allow access. If your SSH public key is at `~/.ssh/id_rsa`, find the fingerprint with,
+DigitalOcean droplets are created with your SSH public key "fingerprint" (i.e. MD5 hash) to allow access. If your SSH public key is at `~/.ssh/id_ed25519.pub`, find the fingerprint with,

 ```bash
 ssh-keygen -E md5 -lf ~/.ssh/id_ed25519.pub | awk '{print $2}'
@ -238,8 +238,8 @@ Digital Ocean requires the SSH public key be uploaded to your account, so you ma
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Droplet type for controllers | "s-2vcpu-2gb" | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
 | worker_type | Droplet type for workers | "s-1vcpu-2gb" | s-1vcpu-2gb, s-2vcpu-2gb, ... |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/) |
+| controller_snippets | Controller Butane snippets | [] | [example](/advanced/customization/) |
+| worker_snippets | Worker Butane snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -52,11 +52,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    google = {
      source = "hashicorp/google"
-      version = "3.75.0"
+      version = "4.3.0"
    }
  }
 }
@ -82,7 +82,7 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count = 2
@ -96,7 +96,7 @@ Reference the [variables docs](#variables) or the [variables.tf](https://github.
 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.

 ```sh
-ssh-add ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_ed25519
 ssh-add -L
 ```

@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.3
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.3
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
@ -186,7 +186,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-
 | region | Google Cloud region | "us-central1" |
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
-| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |

 Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Fedora CoreOS [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep fedora-coreos`.

@ -218,8 +218,8 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "stable", "testing", "next" |
 | disk_size | Size of the disk in GB | 30 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
-| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
-| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
+| controller_snippets | Controller Butane snippets | [] | [examples](/advanced/customization/) |
+| worker_snippets | Worker Butane snippets | [] | [examples](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "cilium" or "flannel" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
--- a/docs/flatcar-linux/aws.md
+++ b/docs/flatcar-linux/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -51,11 +51,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    aws = {
      source = "hashicorp/aws"
-      version = "3.48.0"
+      version = "3.67.0"
    }
  }
 }
@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.0"

  # AWS
  cluster_name = "tempest"
@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.21.3
-ip-10-0-26-65  Ready   <none>  10m  v1.21.3
-ip-10-0-41-21  Ready   <none>  10m  v1.21.3
+ip-10-0-3-155  Ready   <none>  10m  v1.23.0
+ip-10-0-26-65  Ready   <none>  10m  v1.23.0
+ip-10-0-41-21  Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/azure.md
+++ b/docs/flatcar-linux/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -48,11 +48,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    azurerm = {
      source = "hashicorp/azurerm"
-      version = "2.68.0"
+      version = "2.88.1"
    }
  }
 }
@ -75,7 +75,7 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.23.0"

  # Azure
  cluster_name   = "ramius"
@ -149,9 +149,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.21.3
-ramius-worker-000001  Ready   <none>  25m  v1.21.3
-ramius-worker-000002  Ready   <none>  24m  v1.21.3
+ramius-controller-0   Ready   <none>  24m  v1.23.0
+ramius-worker-000001  Ready   <none>  25m  v1.23.0
+ramius-worker-000002  Ready   <none>  24m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/bare-metal.md
+++ b/docs/flatcar-linux/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.21.3 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -138,11 +138,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    matchbox = {
      source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
    }
  }
 }
@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"

  # bare-metal
  cluster_name            = "mercury"
@ -293,9 +293,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.21.3
-node2.example.com   Ready   <none>  10m  v1.21.3
-node3.example.com   Ready   <none>  10m  v1.21.3
+node1.example.com   Ready   <none>  10m  v1.23.0
+node2.example.com   Ready   <none>  10m  v1.23.0
+node3.example.com   Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/digitalocean.md
+++ b/docs/flatcar-linux/digitalocean.md
@ -1,6 +1,6 @@
 # DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -51,7 +51,7 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    digitalocean = {
      source = "digitalocean/digitalocean"
@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.23.0"

  # Digital Ocean
  cluster_name = "nemo"
@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.21.3
-10.132.115.81      Ready   <none>  10m  v1.21.3
-10.132.124.107     Ready   <none>  10m  v1.21.3
+10.132.110.130     Ready   <none>  10m  v1.23.0
+10.132.115.81      Ready   <none>  10m  v1.23.0
+10.132.124.107     Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/google-cloud.md
+++ b/docs/flatcar-linux/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.21.3 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -52,11 +52,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    google = {
      source = "hashicorp/google"
-      version = "3.75.0"
+      version = "4.3.0"
    }
  }
 }
@ -92,7 +92,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.23.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -167,9 +167,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.3
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.3
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@ -57,7 +57,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -66,7 +66,7 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count = 2
@ -95,9 +95,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.21.3
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.21.3
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.21.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/docs/topics/faq.md
+++ b/docs/topics/faq.md
@ -6,10 +6,6 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 Formats rise and evolve. Typhoon may choose to adapt the format over time (with lots of forewarning). However, the authors' have built several Kubernetes "distros" before and learned from mistakes - Terraform modules are the right format for now.

-## Get Help
-
-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
-
 ## Security Issues

 If you find security issues, please see [security disclosures](/topics/security/#disclosures).
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar

 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
  ...
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.21.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"
  ...
 }
 ```
@ -197,7 +197,7 @@ Typhoon modules have been updated for v0.13.x. Poseidon publishes [providers](/t

 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.21.3 - ?       | v0.13.x, v0.14.4+, v0.15.x, v1.0.x |
+| v1.21.2 - ?       | v0.13.x, v0.14.4+, v0.15.x, v1.0.x |
 | v1.21.1 - v1.21.1 | v0.13.x, v0.14.4+, v0.15.x |
 | v1.20.2 - v1.21.0 | v0.13.x, v0.14.4+   |
 | v1.20.0 - v1.20.2 | v0.13.x             |
@ -208,76 +208,3 @@ Typhoon modules have been updated for v0.13.x. Poseidon publishes [providers](/t
 | v1.7.3 - v1.9.1   | v0.10.x             |
 | v1.6.4 - v1.7.2   | v0.9.x              |

-### New Workspace
-
-With a new Terraform workspace, use Terraform v0.15.x and the updated Typhoon [tutorials](/fedora-coreos/aws/#provider).
-
-### Existing Workspace
-
-An existing Terraform workspace may already manage earlier Typhoon clusters created with Terraform v0.12.x.
-
-First, upgrade `terraform-provider-ct` to v0.6.1 following the [guide](#upgrade-terraform-provider-ct) above. As usual, read about how `apply` affects existing cluster nodes when `ct` is upgraded. But `terraform-provider-ct` v0.6.1 is compatible with both Terraform v0.12 and v0.13, so do this first.
-
-```tf
-provider "ct" {
-  version = "0.6.1"
-}
-```
-
-Next, create Typhoon clusters using the `ref` that introduced Terraform v0.13 forward compatibility (`v1.18.8`) or later. You will see a compatibility warning. Use blue/green cluster replacement to shift to these new clusters, then eliminate older clusters.
-
-```
-module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.18.8"
-  ...
-}
-```
-
-Install Terraform v0.13. Once all clusters in a workspace are on `v1.18.8` or above, you are ready to start using Terraform v0.13.
-
-```
-terraform version
-v0.13.0
-```
-
-Update `providers.tf` to match the Typhoon [tutorials](/fedora-coreos/aws/#provider) and use new `required_providers` block.
-
-```
-terraform init
-terraform 0.13upgrade    # sometimes helpful
-```
-
-!!! note
-    You will see `Could not retrieve the list of available versions for provider -/ct: provider`
-
-In state files, existing clusters use Terraform v0.12 providers (e.g. `-/aws`). Pivot to Terraform v0.13 providers (e.g. `hashicorp/aws`) with the following commands, as applicable. Repeat until `terraform init` no longer shows old-style providers.
-
-```
-terraform state replace-provider -- -/aws hashicorp/aws
-terraform state replace-provider -- -/azurerm hashicorp/azurerm
-terraform state replace-provider -- -/google hashicorp/google
-
-terraform state replace-provider -- -/digitalocean digitalocean/digitalocean
-terraform state replace-provider -- -/ct poseidon/ct
-terraform state replace-provider -- -/matchbox poseidon/matchbox
-
-terraform state replace-provider -- -/local hashicorp/local
-terraform state replace-provider -- -/null hashicorp/null
-terraform state replace-provider -- -/random hashicorp/random
-terraform state replace-provider -- -/template hashicorp/template
-terraform state replace-provider -- -/tls hashicorp/tls
-```
-
-Finally, verify Terraform v0.13 plan shows no diff.
-
-```
-terraform plan
-No changes. Infrastructure is up-to-date.
-```
-
-### v0.12.x
-
-Terraform [v0.12](https://www.hashicorp.com/blog/announcing-terraform-0-12) introduced major changes to the provider plugin protocol and HCL language (first-class expressions, formal list and map types, nullable variables, variable constraints, and short-circuiting ternary operators).
-
-Typhoon modules have been adapted for Terraform v0.12. Provider plugins requirements now enforce v0.12 compatibility. However, some HCL language changes were breaking (list [type hint](https://www.terraform.io/upgrade-guides/0-12.html#referring-to-list-variables) workarounds in v0.11 now have new meaning). Typhoon cannot offer both v0.11 and v0.12 compatibility in the same release. Upcoming releases require upgrading Terraform to v0.12.
-
--- a/docs/topics/security.md
+++ b/docs/topics/security.md
@ -62,9 +62,9 @@ The Kubelet image is published to Quay.io and Dockerhub.
 Two tag styles indicate the build strategy used.

 * Typhoon internal infra publishes single and multi-arch images (e.g. `v1.18.4`, `v1.18.4-amd64`, `v1.18.4-arm64`, `v1.18.4-2-g23228e6-amd64`, `v1.18.4-2-g23228e6-arm64`)
-* Quay and Dockerhub automated builds publish verifiable images (e.g. `build-SHA` on Quay, `build-TAG` on Dockerhub)
+* Quay automated builds publish verifiable images (e.g. `build-SHA` on Quay)

-The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay/Dockerhub (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization/#system-images).
+The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization/#system-images).

 ### flannel-cni

--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.21.3 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/Show More
+++ b/Show More