Update Prometheus and Grafana addons

Fix Azure backend_address_pool_id deprecation warning
* Change to `backend_address_pool_ids` list
2025-08-02 06:21:33 +02:00 · 2021-12-15 08:16:46 -08:00 · 2021-12-14 10:26:08 -08:00 · 2021-12-13 17:44:02 -08:00 · 2021-12-10 11:29:49 -08:00 · 2021-12-10 11:26:05 -08:00
101 changed files with 449 additions and 343 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,9 +4,115 @@ Notable changes between versions.

 ## Latest

+* Kubernetes [v1.23.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230)
+* Normalize CA cert mounts in static Pods and kube-proxy ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+* Set Kubelet resolver config to `/run/systemd/resolve/resolv.conf` ([#1082](https://github.com/poseidon/typhoon/pull/1082))
+* Update Cilium from v1.10.5 to [v1.11.0](https://github.com/cilium/cilium/releases/tag/v1.11.0) ([#1083](https://github.com/poseidon/typhoon/pull/1083))
+* With Calico, add missing `caliconodestatuses` CRD ([#289](https://github.com/poseidon/terraform-render-bootstrap/pull/289))
+* Change `enable_aggregation` default to true ([#279](https://github.com/poseidon/terraform-render-bootstrap/pull/279))
+* Remove deprecated `--port` from `kube-scheduler` ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+
+### AWS
+
+* Change controller node default `disk_iops` to 3000 ([#1073](https://github.com/poseidon/typhoon/pull/1073))
+
+### Azure
+
+* Fix warning about deprecated `backend_address_pool_id` ([#1086](https://github.com/poseidon/typhoon/pull/1086))
+
+### Fedora CoreOS
+
+* Fix Fedora ARM64 workers to official Fedora CoreOS AMIs ([#1072](https://github.com/poseidon/typhoon/pull/1072))
+  * Should have been changed alongside controller AMIs in ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+  * Old Poseidon built ARM64 AMIs have been deleted
+
+### Addons
+
+* Update nginx-ingress from v1.0.5 to [v1.1.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.0)
+* Update Prometheus from v2.31.1 to [v2.32.0](https://github.com/prometheus/prometheus/releases/tag/v2.32.0)
+* Update kube-state-metrics from v2.2.4 to [v2.3.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.3.0)
+* Update node-exporter from v1.3.0 to [v1.3.1](https://github.com/prometheus/node_exporter/releases/tag/v1.3.1)
+* Update Grafana from v8.2.4 to [v8.3.3](https://github.com/grafana/grafana/releases/tag/v8.3.3)
+
+### Known Issues
+
+* Calico does not yet support Kubernetes v1.23.0, use `flannel` or `cilium` ([calico#5011](https://github.com/projectcalico/calico/issues/5011))
+
+## v1.22.4
+
+* Kubernetes [v1.22.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224)
+* Update CoreDNS from v1.8.4 to [v1.8.6](https://github.com/poseidon/terraform-render-bootstrap/pull/284)
+* Update Calico from v3.20.2 to [v3.21.0](https://github.com/projectcalico/calico/releases/tag/v3.21.0)
+* Update flannel from v0.14.0 to [v0.15.1](https://github.com/flannel-io/flannel/releases/tag/v0.15.1)
+
+### Google
+
+* Allow use of Terraform provider `google` [v4.0+](https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0)
+
+### Flatcar Linux
+
+* Change Kubelet mounts for cgroups v2 ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+* Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+
+### Addons
+
+* Update Prometheus from v2.30.3 to [v2.31.1](https://github.com/prometheus/prometheus/releases/tag/v2.31.1)
+* Update node-exporter from v1.2.2 to [v1.3.0](https://github.com/prometheus/node_exporter/releases/tag/v1.3.0)
+* Update kube-state-metrics from v2.2.3 to [v2.2.4](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.4)
+* Update Grafana from v8.2.1 to [v8.2.4](https://github.com/grafana/grafana/releases/tag/v8.2.4)
+* Update nginx-ingress from v1.0.4 to [v1.0.5](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.5)
+
+## v1.23.3
+
+* Kubernetes [v1.22.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223)
+* Update etcd from v3.5.0 to [v3.5.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.1)
+* Update Cilium from v1.10.4 to [v1.10.5](https://github.com/cilium/cilium/releases/tag/v1.10.5)
+* Update Calico from v3.20.1 to [v3.20.2](https://github.com/projectcalico/calico/releases/tag/v3.20.2)
+  * Use Calico's iptables legacy vs nft auto-detection
+* Update flannel from v0.13.0 to v0.14.0
+
+### Bare-Metal
+
+* Require Terraform provider `poseidon/matchbox` v0.5+ ([#1048](https://github.com/poseidon/typhoon/pull/1048))
+
+### Addons
+
+* Update nginx-ingress from v1.0.0 to [v1.0.4](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4)
+* Update Prometheus from v2.29.2 to [v2.30.3](https://github.com/prometheus/prometheus/releases/tag/v2.30.3)
+* Update kube-state-metrics from v2.2.0 to [v2.2.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.3)
+* Update Grafana from v8.1.2 to [v8.2.1](https://github.com/grafana/grafana/releases/tag/v8.2.1)
+
+## v1.22.2
+
+* Kubernetes [v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222)
+* Update Cilium from v1.10.3 to [v1.10.4](https://github.com/cilium/cilium/releases/tag/v1.10.4)
+* Update Calico from v3.20.0 to [v3.20.1](https://github.com/projectcalico/calico/releases/tag/v3.20.1)
+* Fix access to ClusterIP services with Cilium ([#276](https://github.com/poseidon/terraform-render-bootstrap/pull/276))
+
+### Fedora CoreOS
+
+* Use Fedora CoreOS ARM64 AMIs ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+
+### Addons
+
+* Update Prometheus from v2.29.1 to [v2.29.2](https://github.com/prometheus/prometheus/releases/tag/v2.29.2)
+* Update kube-state-metrics from v2.1.1 to [v2.2.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.0)
+
+## v1.22.1
+
+* Kubernetes [v1.22.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221)
+* Update Calico from v3.19.1 to [v3.20.0](https://github.com/projectcalico/calico/releases/tag/v3.20.0)
+
+### Addons
+
+* Update nginx-ingress from v1.0.0-beta.1 to [v1.0.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0)
+* Update Prometheus from v2.28.1 to [v2.29.1](https://github.com/prometheus/prometheus/releases/tag/v2.29.1)
+* Update Grafana from v8.1.1 to [v8.1.2](https://github.com/grafana/grafana/releases/tag/v8.1.2)
+
 ## v1.22.0

 * Kubernetes [v1.22.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1220)
+* Update etcd from v3.4.16 to [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0)
 * Switch `kube-controller-manager` and `kube-scheduler` to use secure port only
  * Update Prometheus config to discover endpoints and use a bearer token to scrape

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@ -58,7 +58,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -97,9 +97,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -24,7 +24,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:8.1.1
+          image: docker.io/grafana/grafana:8.3.3
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -23,7 +23,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -21,7 +21,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.28.1
+          image: quay.io/prometheus/prometheus:v2.32.0
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -25,7 +25,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.1
+        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -28,7 +28,7 @@ spec:
      hostPID: true
      containers:
      - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.2.2
+        image: quay.io/prometheus/node-exporter:v1.3.1
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
  }
 }

-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
  count = var.arch == "arm64" ? 1 : 0

  most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]

  filter {
    name   = "architecture"
@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
  }

  filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
-
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -13,7 +13,5 @@ module "bootstrap" {
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
  daemonset_tolerations = var.daemonset_tolerations
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -54,7 +54,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -97,6 +97,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -122,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -218,6 +219,7 @@ storage:
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
          ETCD_UNSUPPORTED_ARCH=arm64
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -66,8 +66,8 @@ variable "disk_type" {

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -142,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/aws/fedora-coreos/kubernetes/versions.tf
+++ b/aws/fedora-coreos/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
  }
 }

-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
  count = var.arch == "arm64" ? 1 : 0

  most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]

  filter {
    name   = "architecture"
@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
  }

  filter {
-    name   = "name"
-    values = ["fedora-coreos-*"]
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
-
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -76,6 +76,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -90,7 +91,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -129,6 +130,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/workers/versions.tf
+++ b/aws/fedora-coreos/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/aws/flatcar-linux/kubernetes/README.md
+++ b/aws/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/flatcar-linux/kubernetes/bootstrap.tf
+++ b/aws/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/aws/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -57,7 +57,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -76,8 +76,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -88,6 +87,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -98,6 +98,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -119,7 +120,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/aws/flatcar-linux/kubernetes/variables.tf
+++ b/aws/flatcar-linux/kubernetes/variables.tf
@ -66,8 +66,8 @@ variable "disk_type" {

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -142,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/aws/flatcar-linux/kubernetes/versions.tf
+++ b/aws/flatcar-linux/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -29,7 +29,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -51,8 +51,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -63,6 +62,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -79,6 +79,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -94,7 +95,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/aws/flatcar-linux/kubernetes/workers/versions.tf
+++ b/aws/flatcar-linux/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -19,8 +19,5 @@ module "bootstrap" {
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
  daemonset_tolerations = var.daemonset_tolerations
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -51,7 +51,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -92,6 +92,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -117,7 +118,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -212,6 +213,7 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/azure/fedora-coreos/kubernetes/lb.tf
+++ b/azure/fedora-coreos/kubernetes/lb.tf
@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
--- a/azure/fedora-coreos/kubernetes/variables.tf
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@ -117,8 +117,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -24,7 +24,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -71,6 +71,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -85,7 +86,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -124,6 +125,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/azure/fedora-coreos/kubernetes/workers/versions.tf
+++ b/azure/fedora-coreos/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/azure/flatcar-linux/kubernetes/README.md
+++ b/azure/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/flatcar-linux/kubernetes/bootstrap.tf
+++ b/azure/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/azure/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -55,7 +55,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -73,8 +73,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -85,6 +84,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -94,6 +94,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -115,7 +116,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/azure/flatcar-linux/kubernetes/lb.tf
+++ b/azure/flatcar-linux/kubernetes/lb.tf
@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
--- a/azure/flatcar-linux/kubernetes/variables.tf
+++ b/azure/flatcar-linux/kubernetes/variables.tf
@ -123,8 +123,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/azure/flatcar-linux/kubernetes/versions.tf
+++ b/azure/flatcar-linux/kubernetes/versions.tf
@ -4,8 +4,8 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -48,8 +48,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -60,6 +59,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -75,6 +75,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -90,7 +91,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/azure/flatcar-linux/kubernetes/workers/versions.tf
+++ b/azure/flatcar-linux/kubernetes/workers/versions.tf
@ -4,7 +4,7 @@ terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
    azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
@ -13,8 +13,6 @@ module "bootstrap" {
  cluster_domain_suffix           = var.cluster_domain_suffix
  enable_reporting                = var.enable_reporting
  enable_aggregation              = var.enable_aggregation
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }


--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -50,7 +50,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -92,6 +92,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -119,7 +120,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
@ -222,6 +223,7 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -23,7 +23,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -71,6 +71,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -120,6 +121,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -44,7 +44,7 @@ resource "matchbox_profile" "controllers" {

  kernel = local.kernel
  initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.controller-ignitions.*.rendered[count.index]
 }
@ -78,7 +78,7 @@ resource "matchbox_profile" "workers" {

  kernel = local.kernel
  initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)

  raw_ignition = data.ct_config.worker-ignitions.*.rendered[count.index]
 }
--- a/bare-metal/fedora-coreos/kubernetes/variables.tf
+++ b/bare-metal/fedora-coreos/kubernetes/variables.tf
@ -146,8 +146,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/bare-metal/fedora-coreos/kubernetes/versions.tf
+++ b/bare-metal/fedora-coreos/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
@ -13,7 +13,7 @@ terraform {

    matchbox = {
      source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
    }
  }
 }
--- a/bare-metal/flatcar-linux/kubernetes/README.md
+++ b/bare-metal/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -63,7 +63,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -81,8 +81,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -93,6 +92,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -103,6 +103,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -124,7 +125,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
@ -35,7 +35,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -56,8 +56,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -68,6 +67,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -84,6 +84,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
--- a/bare-metal/flatcar-linux/kubernetes/variables.tf
+++ b/bare-metal/flatcar-linux/kubernetes/variables.tf
@ -151,8 +151,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/bare-metal/flatcar-linux/kubernetes/versions.tf
+++ b/bare-metal/flatcar-linux/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
@ -13,7 +13,7 @@ terraform {

    matchbox = {
      source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
    }
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -17,8 +17,5 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }

--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -52,7 +52,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -95,6 +95,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -129,7 +130,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -219,3 +220,5 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -26,7 +26,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -69,6 +69,7 @@ systemd:
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -92,7 +93,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -126,3 +127,4 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
--- a/digital-ocean/fedora-coreos/kubernetes/variables.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/variables.tf
@ -94,8 +94,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/digital-ocean/fedora-coreos/kubernetes/versions.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/digital-ocean/flatcar-linux/kubernetes/README.md
+++ b/digital-ocean/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -65,7 +65,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -84,8 +84,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -96,6 +95,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -106,6 +106,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -127,7 +128,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
@ -37,7 +37,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -59,8 +59,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -71,6 +70,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -81,6 +81,7 @@ systemd:
          --node-labels=node.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -96,7 +97,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/digital-ocean/flatcar-linux/kubernetes/variables.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/variables.tf
@ -94,8 +94,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 # unofficial, undocumented, unsupported
--- a/digital-ocean/flatcar-linux/kubernetes/versions.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/docs/advanced/arm64.md
+++ b/docs/advanced/arm64.md
@ -8,20 +8,13 @@ Typhoon has experimental support for ARM64 with Fedora CoreOS on AWS. Full clust
 !!! note
    Currently, CNI networking must be set to flannel or Cilium.

-## AMIs
-
-In lieu of official Fedora CoreOS ARM64 AMIs, Poseidon publishes experimental ARM64 AMIs to a few regions (us-east-1, us-east-2, us-west-1). These AMIs may be **removed** at any time and will be replaced when Fedora CoreOS publishes equivalents.
-
-!!! note
-    AMIs are only published to a few regions, and AWS availability of ARM instance types varies.
-
 ## Cluster

 Create a cluster with ARM64 controller and worker nodes. Container workloads must be `arm64` compatible and use `arm64` container images.

 ```tf
 module "gravitas" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"

  # AWS
  cluster_name = "gravitas"
@ -47,9 +40,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes.
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-12-178   Ready    <none>   101s   v1.22.0   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-18-93    Ready    <none>   102s   v1.22.0   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-90-10    Ready    <none>   104s   v1.22.0   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-12-178   Ready    <none>   101s   v1.23.0   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-18-93    Ready    <none>   102s   v1.23.0   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-90-10    Ready    <none>   104s   v1.23.0   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
 ```

 ## Hybrid
@ -60,7 +53,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo

    ```tf
    module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"

      # AWS
      cluster_name = "gravitas"
@ -83,7 +76,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo

    ```tf
    module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # AWS
      vpc_id          = module.gravitas.vpc_id
@ -108,9 +101,9 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 ```
 $ kubectl get nodes -o wide
 NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION             CONTAINER-RUNTIME
-ip-10-0-1-81    Ready    <none>   4m28s   v1.22.0   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-17-86   Ready    <none>   4m28s   v1.22.0   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
-ip-10-0-21-45   Ready    <none>   4m28s   v1.22.0   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-40-36   Ready    <none>   4m22s   v1.22.0   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-1-81    Ready    <none>   4m28s   v1.23.0   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-17-86   Ready    <none>   4m28s   v1.23.0   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
+ip-10-0-21-45   Ready    <none>   4m28s   v1.23.0   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-40-36   Ready    <none>   4m22s   v1.23.0   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
 ```

--- a/docs/advanced/nodes.md
+++ b/docs/advanced/nodes.md
@ -36,7 +36,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to

    ```tf
    module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

      # Google Cloud
      cluster_name  = "yavin"
@ -57,7 +57,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to

    ```tf
    module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      cluster_name = "yavin"
@ -89,7 +89,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and

    ```tf
    module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

      # Google Cloud
      cluster_name  = "yavin"
@ -110,7 +110,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and

    ```tf
    module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      cluster_name = "yavin"
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).

    ```tf
    module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # AWS
      vpc_id          = module.tempest.vpc_id
@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).

    ```tf
    module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.23.0"

      # AWS
      vpc_id          = module.tempest.vpc_id
@ -111,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste

    ```tf
    module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Azure
      region                  = module.ramius.region
@ -137,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste

    ```tf
    module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.23.0"

      # Azure
      region                  = module.ramius.region
@ -207,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c

    ```tf
    module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      region       = "europe-west2"
@ -231,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c

    ```tf
    module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.23.0"

      # Google Cloud
      region       = "europe-west2"
@ -262,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.22.0
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.22.0
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.22.0
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.22.0
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.22.0
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.23.0
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.23.0
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.23.0
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.23.0
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.23.0
 ```

 ### Variables
--- a/docs/architecture/operating-systems.md
+++ b/docs/architecture/operating-systems.md
@ -15,24 +15,26 @@ Together, they diversify Typhoon to support a range of container technologies.
 ## Host Properties

 | Property          | Flatcar Linux | Fedora CoreOS |
-|-------------------|---------------------------------|---------------|
-| Kernel            | ~5.10.x | ~5.12.x |
-| systemd           | 247 | 248 |
+|-------------------|---------------|---------------|
+| Kernel            | ~5.10.x       | ~5.14.x       |
+| systemd           | 247           | 249           |
+| Username          | core          | core          |
 | Ignition system   | Ignition v2.x spec | Ignition v3.x spec |
-| Container Engine  | docker 19.3.15  | docker 20.10.6 |
 | storage driver    | overlay2 (extfs)  | overlay2 (xfs) |
-| logging driver    | json-file | journald |
-| cgroup driver     | cgroupfs (except Flatcar edge) | systemd  |
-| Networking        | systemd-networkd | NetworkManager |
-| Username          | core      | core |
+| logging driver    | json-file     | journald      |
+| cgroup driver     | systemd       | systemd       |
+| cgroup version    | v2            | v2            |
+| Networking        | systemd-networkd | NetworkManager   |
+| Resolver          | systemd-resolved | systemd-resolved |

 ## Kubernetes Properties

 | Property          | Flatcar Linux | Fedora CoreOS |
-|-------------------|-----------------|---------------|
+|-------------------|---------------|---------------|
 | single-master     | all platforms | all platforms |
 | multi-master      | all platforms | all platforms |
 | control plane     | static pods   | static pods   |
+| Container Runtime | docker 20.10  | docker 20.10  |
 | kubelet image     | kubelet [image](https://github.com/poseidon/kubelet) with upstream binary | kubelet [image](https://github.com/poseidon/kubelet) with upstream binary |
 | control plane images | upstream images | upstream images |
 | on-host etcd      | docker    | podman |
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -51,11 +51,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    aws = {
      source = "hashicorp/aws"
-      version = "3.48.0"
+      version = "3.67.0"
    }
  }
 }
@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"

  # AWS
  cluster_name = "tempest"
@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.22.0
-ip-10-0-26-65  Ready   <none>   10m  v1.22.0
-ip-10-0-41-21  Ready   <none>   10m  v1.22.0
+ip-10-0-3-155  Ready   <none>   10m  v1.23.0
+ip-10-0-26-65  Ready   <none>   10m  v1.23.0
+ip-10-0-41-21  Ready   <none>   10m  v1.23.0
 ```

 List the pods.
--- a/docs/fedora-coreos/azure.md
+++ b/docs/fedora-coreos/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -48,11 +48,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    azurerm = {
      source = "hashicorp/azurerm"
-      version = "2.68.0"
+      version = "2.88.1"
    }
  }
 }
@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.23.0"

  # Azure
  cluster_name   = "ramius"
@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.22.0
-ramius-worker-000001  Ready   <none>  25m  v1.22.0
-ramius-worker-000002  Ready   <none>  24m  v1.22.0
+ramius-controller-0   Ready   <none>  24m  v1.23.0
+ramius-worker-000001  Ready   <none>  25m  v1.23.0
+ramius-worker-000002  Ready   <none>  24m  v1.23.0
 ```

 List the pods.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.22.0 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -138,11 +138,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    matchbox = {
      source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
    }
  }
 }
@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.23.0"

  # bare-metal
  cluster_name            = "mercury"
@ -283,9 +283,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.22.0
-node2.example.com   Ready   <none>  10m  v1.22.0
-node3.example.com   Ready   <none>  10m  v1.22.0
+node1.example.com   Ready   <none>  10m  v1.23.0
+node2.example.com   Ready   <none>  10m  v1.23.0
+node3.example.com   Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@ -1,6 +1,6 @@
 # DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -51,7 +51,7 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    digitalocean = {
      source = "digitalocean/digitalocean"
@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.23.0"

  # Digital Ocean
  cluster_name = "nemo"
@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.22.0
-10.132.115.81      Ready   <none>  10m  v1.22.0
-10.132.124.107     Ready   <none>  10m  v1.22.0
+10.132.110.130     Ready   <none>  10m  v1.23.0
+10.132.115.81      Ready   <none>  10m  v1.23.0
+10.132.124.107     Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -52,11 +52,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    google = {
      source = "hashicorp/google"
-      version = "3.75.0"
+      version = "4.3.0"
    }
  }
 }
@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/aws.md
+++ b/docs/flatcar-linux/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -51,11 +51,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    aws = {
      source = "hashicorp/aws"
-      version = "3.48.0"
+      version = "3.67.0"
    }
  }
 }
@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.0"

  # AWS
  cluster_name = "tempest"
@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.22.0
-ip-10-0-26-65  Ready   <none>  10m  v1.22.0
-ip-10-0-41-21  Ready   <none>  10m  v1.22.0
+ip-10-0-3-155  Ready   <none>  10m  v1.23.0
+ip-10-0-26-65  Ready   <none>  10m  v1.23.0
+ip-10-0-41-21  Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/azure.md
+++ b/docs/flatcar-linux/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -48,11 +48,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    azurerm = {
      source = "hashicorp/azurerm"
-      version = "2.68.0"
+      version = "2.88.1"
    }
  }
 }
@ -75,7 +75,7 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.23.0"

  # Azure
  cluster_name   = "ramius"
@ -149,9 +149,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.22.0
-ramius-worker-000001  Ready   <none>  25m  v1.22.0
-ramius-worker-000002  Ready   <none>  24m  v1.22.0
+ramius-controller-0   Ready   <none>  24m  v1.23.0
+ramius-worker-000001  Ready   <none>  25m  v1.23.0
+ramius-worker-000002  Ready   <none>  24m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/bare-metal.md
+++ b/docs/flatcar-linux/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.22.0 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -138,11 +138,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    matchbox = {
      source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
    }
  }
 }
@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"

  # bare-metal
  cluster_name            = "mercury"
@ -293,9 +293,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.22.0
-node2.example.com   Ready   <none>  10m  v1.22.0
-node3.example.com   Ready   <none>  10m  v1.22.0
+node1.example.com   Ready   <none>  10m  v1.23.0
+node2.example.com   Ready   <none>  10m  v1.23.0
+node3.example.com   Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/digitalocean.md
+++ b/docs/flatcar-linux/digitalocean.md
@ -1,6 +1,6 @@
 # DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -51,7 +51,7 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    digitalocean = {
      source = "digitalocean/digitalocean"
@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.23.0"

  # Digital Ocean
  cluster_name = "nemo"
@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.22.0
-10.132.115.81      Ready   <none>  10m  v1.22.0
-10.132.124.107     Ready   <none>  10m  v1.22.0
+10.132.110.130     Ready   <none>  10m  v1.23.0
+10.132.115.81      Ready   <none>  10m  v1.23.0
+10.132.124.107     Ready   <none>  10m  v1.23.0
 ```

 List the pods.
--- a/docs/flatcar-linux/google-cloud.md
+++ b/docs/flatcar-linux/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -52,11 +52,11 @@ terraform {
  required_providers {
    ct = {
      source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
    }
    google = {
      source = "hashicorp/google"
-      version = "3.75.0"
+      version = "4.3.0"
    }
  }
 }
@ -92,7 +92,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.23.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -167,9 +167,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@ -57,7 +57,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -95,9 +95,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```

 List the pods.
--- a/docs/topics/faq.md
+++ b/docs/topics/faq.md
@ -6,10 +6,6 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 Formats rise and evolve. Typhoon may choose to adapt the format over time (with lots of forewarning). However, the authors' have built several Kubernetes "distros" before and learned from mistakes - Terraform modules are the right format for now.

-## Get Help
-
-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
-
 ## Security Issues

 If you find security issues, please see [security disclosures](/topics/security/#disclosures).
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar

 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
  ...
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"
  ...
 }
 ```
--- a/docs/topics/security.md
+++ b/docs/topics/security.md
@ -62,9 +62,9 @@ The Kubelet image is published to Quay.io and Dockerhub.
 Two tag styles indicate the build strategy used.

 * Typhoon internal infra publishes single and multi-arch images (e.g. `v1.18.4`, `v1.18.4-amd64`, `v1.18.4-arm64`, `v1.18.4-2-g23228e6-amd64`, `v1.18.4-2-g23228e6-arm64`)
-* Quay and Dockerhub automated builds publish verifiable images (e.g. `build-SHA` on Quay, `build-TAG` on Dockerhub)
+* Quay automated builds publish verifiable images (e.g. `build-SHA` on Quay)

-The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay/Dockerhub (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization/#system-images).
+The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization/#system-images).

 ### flannel-cni

--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
+++ b/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@ -14,8 +14,6 @@ module "bootstrap" {
  enable_aggregation    = var.enable_aggregation
  daemonset_tolerations = var.daemonset_tolerations

-  trusted_certs_dir = "/etc/pki/tls/certs"
-
  // temporary
  external_apiserver_port = 443
 }
--- a/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
@ -12,7 +12,7 @@ systemd:
        Wants=network-online.target network.target
        After=network-online.target
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        Type=exec
        ExecStartPre=/bin/mkdir -p /var/lib/etcd
        ExecStartPre=-/usr/bin/podman rm etcd
@ -51,7 +51,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -92,6 +92,7 @@ systemd:
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@ -117,7 +118,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -212,6 +213,7 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/google-cloud/fedora-coreos/kubernetes/variables.tf
+++ b/google-cloud/fedora-coreos/kubernetes/variables.tf
@ -118,8 +118,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/google-cloud/fedora-coreos/kubernetes/versions.tf
+++ b/google-cloud/fedora-coreos/kubernetes/versions.tf
@ -3,9 +3,9 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    google   = ">= 2.19, < 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    google   = ">= 2.19, < 5.0"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -24,7 +24,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -71,6 +71,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/podman stop kubelet
@ -85,7 +86,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
@ -124,6 +125,7 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
  users:
    - name: core
--- a/google-cloud/fedora-coreos/kubernetes/workers/versions.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    google   = ">= 2.19, < 4.0"
-    template = "~> 2.1"
+    google   = ">= 2.19, < 5.0"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/google-cloud/flatcar-linux/kubernetes/README.md
+++ b/google-cloud/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/google-cloud/flatcar-linux/kubernetes/bootstrap.tf
+++ b/google-cloud/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/google-cloud/flatcar-linux/kubernetes/cl/controller.yaml
@ -10,7 +10,7 @@ systemd:
        Requires=docker.service
        After=docker.service
        [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
        ExecStartPre=/usr/bin/docker run -d \
          --name etcd \
          --network host \
@ -55,7 +55,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -73,8 +73,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -85,6 +84,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -95,6 +95,7 @@ systemd:
          --pod-manifest-path=/etc/kubernetes/manifests \
          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -115,7 +116,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/google-cloud/flatcar-linux/kubernetes/variables.tf
+++ b/google-cloud/flatcar-linux/kubernetes/variables.tf
@ -112,8 +112,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
--- a/google-cloud/flatcar-linux/kubernetes/versions.tf
+++ b/google-cloud/flatcar-linux/kubernetes/versions.tf
@ -3,9 +3,9 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    google   = ">= 2.19, < 4.0"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    google   = ">= 2.19, < 5.0"
+    template = "~> 2.2"
+    null     = ">= 2.1"

    ct = {
      source  = "poseidon/ct"
--- a/google-cloud/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/google-cloud/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -48,8 +48,7 @@ systemd:
          -v /usr/lib/os-release:/etc/os-release:ro \
          -v /lib/modules:/lib/modules:ro \
          -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
          -v /var/lib/docker:/var/lib/docker \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@ -60,6 +59,7 @@ systemd:
          --authentication-token-webhook \
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -75,6 +75,7 @@ systemd:
          %{~ endfor ~}
          --pod-manifest-path=/etc/kubernetes/manifests \
          --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
          --rotate-certificates \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStart=docker logs -f kubelet
@ -90,7 +91,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/google-cloud/flatcar-linux/kubernetes/workers/versions.tf
+++ b/google-cloud/flatcar-linux/kubernetes/workers/versions.tf
@ -3,8 +3,8 @@
 terraform {
  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    google   = ">= 2.19, < 4.0"
-    template = "~> 2.1"
+    google   = ">= 2.19, < 5.0"
+    template = "~> 2.2"

    ct = {
      source  = "poseidon/ct"
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Dalton Hubble	bb5e5811ec	Update Prometheus and Grafana addons	2021-12-15 08:16:46 -08:00
Dalton Hubble	16aa997604	Fix Azure `backend_address_pool_id` deprecation warning * Change to `backend_address_pool_ids` list	2021-12-14 10:26:08 -08:00
dependabot[bot]	fb6650b06b	Bump mkdocs-material from 8.0.4 to 8.1.1 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 8.0.4 to 8.1.1. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/8.0.4...8.1.1) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2021-12-13 17:44:02 -08:00
Dalton Hubble	43c6558aaf	Update nginx-ingress and monitoring addons	2021-12-10 11:29:49 -08:00
Dalton Hubble	125008fbb3	Update Cilium from v1.10.5 to v1.11.0 * https://github.com/cilium/cilium/releases/tag/v1.11.0	2021-12-10 11:26:05 -08:00
Dalton Hubble	136107b448	Set Kubelet resolver config to /run/systemd/resolve/resolv.conf * Both Flatcar Linux and Fedora CoreOS use systemd-resolved, but they setup /etc/resolv.conf symlinks differently * Prefer using /run/systemd/resolve/resolv.conf directly, which also updates to reflect runtime changes (e.g. resolvectl)	2021-12-10 08:22:30 -08:00
Dalton Hubble	e97c1cc9e5	Enable Kubernetes aggregation by default * Change `enable_aggregation` default from false to true * These days, Kubernetes control plane components emit annoying messages related to assumptions baked into the Kubernetes API Aggregation Layer if you don't enable it. Further the conformance tests force you to remember to enable it if you care about passing those * This change is motivated by eliminating annoyances, rather than any enthusiasm for Kubernetes' aggregation features Rel: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/	2021-12-09 17:30:35 -08:00
Dalton Hubble	39da5b53f5	Update operating system notes in architecture docs	2021-12-09 17:21:24 -08:00
Dalton Hubble	41f739891b	Normalize CA certs mounts in static Pods and kube-proxy * Mount both /etc/ssl/certs and /etc/pki into control plane static pods and kube-proxy, rather than choosing one based a variable (set based on Flatcar Linux or Fedora CoreOS) * Remove deprecated `--port` from `kube-scheduler` static Pod	2021-12-09 09:56:37 -08:00
Dalton Hubble	861021ee98	Update Kubernetes from v1.22.4 to v1.23.0 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230 * With Calico, add missing caliconodestatuses CRD added in v3.21.0 https://github.com/poseidon/terraform-render-bootstrap/pull/289	2021-12-09 09:28:41 -08:00
Dalton Hubble	9d583ab377	Fix null provider version constraint on Google Cloud * Part of https://github.com/poseidon/typhoon/pull/1074	2021-12-08 14:06:38 -08:00
Dalton Hubble	c1d28e6f61	Change default disk_iops on Flatcar Linux * Same as #1073, but for Flatcar Linux on AWS as well	2021-12-07 16:52:55 -08:00
Dalton Hubble	a8fd21d250	Update minimum Terraform provider versions * Update `null` provider to allow use of v3.1.x releases, instead of being stuck on v2.1.2 * Update min versions in terraform-render-boostrap https://github.com/poseidon/terraform-render-bootstrap/pull/287 * Document the recommended versions of Terraform cloud providers	2021-12-07 16:26:34 -08:00
Dalton Hubble	9c626c9dbd	Change default `disk_iops` from unset to 3000 * Since v1.21.3 switched controllers default disk type from `gp2` to `gp3`, an iops diff has been shown (harmless, but annoying) * Controller nodes default to a 30GB `gp3` disk. `gp3` disks do respect `iops` and the corresponding default is 3000	2021-12-07 15:44:09 -08:00
Dalton Hubble	85252dec6e	Switch FCOS workers to official Fedora CoreOS AMIs * Fix worker nodes to use official Fedora CoreOS AMIs, instead of the older Poseidon built AMIs (now removed). This should have been part of #1038, but was missed in code review * Poseidon build AMIs have been deleted (so I don't have to keep paying to host them for people)	2021-12-07 15:31:47 -08:00
dependabot[bot]	298ea65d3e	Bump mkdocs-material from 8.0.3 to 8.0.4 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 8.0.3 to 8.0.4. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/8.0.3...8.0.4) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-12-07 15:29:00 -08:00
dependabot[bot]	c0ab15ba22	Bump mkdocs-material from 7.3.6 to 8.0.3 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.3.6 to 8.0.3. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Upgrade guide](https://github.com/squidfunk/mkdocs-material/blob/master/docs/upgrade.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.3.6...8.0.3) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>	2021-12-02 15:25:40 -08:00
Dalton Hubble	5d7b6f611e	Update nginx-ingess and Prometheus exporter addons	2021-11-21 09:28:17 -08:00
Dalton Hubble	93594292eb	Update Kubernetes from v1.22.3 to v1.22.4 * Update flannel from v0.15.0 to v0.15.1 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224	2021-11-17 19:53:32 -08:00
dependabot[bot]	0546608e77	Bump pymdown-extensions from 9.0 to 9.1 Bumps [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) from 9.0 to 9.1. - [Release notes](https://github.com/facelessuser/pymdown-extensions/releases) - [Commits](https://github.com/facelessuser/pymdown-extensions/compare/9.0...9.1) --- updated-dependencies: - dependency-name: pymdown-extensions dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2021-11-17 18:41:22 -08:00
Dalton Hubble	94b2793e40	Update CoreDNS from v1.8.4 to v1.8.6 * https://coredns.io/2021/10/07/coredns-1.8.6-release/	2021-11-12 21:09:04 -08:00
Dalton Hubble	4fd43b39ad	Fix Flatcar Linux docker driver and add cgroups v2 * Remove `/sys/fs/cgroup/systemd` mount since Flatcar Linux uses cgroups v2 * Flatcar Linux's `docker` switched from the `cgroupfs` to `systemd` driver without notice	2021-11-12 21:07:20 -08:00
Dalton Hubble	65083aca7d	Update Calico and Flannel CNI providers * Update Calico from v3.20.2 to v3.21.0 * Update Flannel from v0.14.0 to v0.15.0	2021-11-12 11:03:39 -08:00
Dalton Hubble	07db4c1143	Allow use of google Terraform provider v4.0+ * https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0	2021-11-11 10:17:58 -08:00
dependabot[bot]	e5d0ce5fd7	Bump mkdocs-material from 7.3.4 to 7.3.6 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.3.4 to 7.3.6. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.3.4...7.3.6) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-11-07 17:01:41 -08:00
Dalton Hubble	b934a13605	Update Prometheus and Grafana addons	2021-11-07 17:00:40 -08:00
Dalton Hubble	cd005a0b27	Prepare for v1.22.3 release	2021-10-28 11:58:55 -07:00
Dalton Hubble	dd4a5a4e7e	Update Kubernetes from v1.22.2 to v1.22.3 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223	2021-10-28 10:11:06 -07:00
Dalton Hubble	af835f976f	Update flannel from v0.13.0 to v0.14.0 * https://github.com/flannel-io/flannel/releases/tag/v0.14.0	2021-10-28 10:09:06 -07:00
dependabot[bot]	9e4a369f76	Bump mkdocs-material from 7.3.3 to 7.3.4 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.3.3 to 7.3.4. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.3.3...7.3.4) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-10-23 10:45:49 -07:00
dependabot[bot]	831d897533	Bump mkdocs from 1.2.2 to 1.2.3 Bumps [mkdocs](https://github.com/mkdocs/mkdocs) from 1.2.2 to 1.2.3. - [Release notes](https://github.com/mkdocs/mkdocs/releases) - [Commits](https://github.com/mkdocs/mkdocs/compare/1.2.2...1.2.3) --- updated-dependencies: - dependency-name: mkdocs dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-10-23 10:31:55 -07:00
Dalton Hubble	17dce49982	Update etcd from v3.5.0 to v3.5.1 * https://github.com/etcd-io/etcd/releases/tag/v3.5.1	2021-10-17 11:28:27 -07:00
Dalton Hubble	5744e10329	Update Cilium from v1.0.4 to v1.0.5 * https://github.com/cilium/cilium/releases/tag/v1.10.5	2021-10-17 11:26:59 -07:00
Dalton Hubble	20748536df	Update nginx-ingress from v1.0.2 to v1.0.4 * https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4	2021-10-17 11:17:43 -07:00
Dalton Hubble	f2e6256dd9	Update Prometheus, kube-state-metrics, and Grafana * Update monitoring addons	2021-10-17 11:15:39 -07:00
Dalton Hubble	443bd5a26b	Add file to hold nodes on iptables-legacy * Add `/etc/fedora-coreos/iptables-legacy.stamp` to declare that `iptables-legacy` should be used instead of `iptables-nft` (until support is added in future releases) * https://github.com/coreos/fedora-coreos-tracker/issues/676	2021-10-11 20:30:49 -07:00
Dalton Hubble	f8162b9be3	Update Calico from v3.20.1 to v3.20.2 * Use Calico's iptables legacy vs nft auto-detection	2021-10-11 20:28:48 -07:00
dependabot[bot]	20ffbba4bf	Bump mkdocs-material from 7.3.1 to 7.3.3 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.3.1 to 7.3.3. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.3.1...7.3.3) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-10-11 19:31:10 -07:00
Dalton Hubble	15117fb95b	Update Prometheus and nginx-ingress	2021-10-05 19:15:58 -07:00
dependabot[bot]	10af8b4120	Bump mkdocs-material from 7.3.0 to 7.3.1 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.3.0 to 7.3.1. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.3.0...7.3.1) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-10-04 20:39:01 -07:00
dependabot[bot]	e51b2903c1	Bump pymdown-extensions from 8.2 to 9.0 Bumps [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) from 8.2 to 9.0. - [Release notes](https://github.com/facelessuser/pymdown-extensions/releases) - [Commits](https://github.com/facelessuser/pymdown-extensions/compare/8.2...9.0) --- updated-dependencies: - dependency-name: pymdown-extensions dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>	2021-10-04 20:38:46 -07:00
Dalton Hubble	cb72b261c7	Update Terraform provider poseidon/matchbox to v0.5+ * Relax version constraint to allow future minor version releases to be used without a corresponding Typhoon change	2021-09-29 23:41:44 -07:00
Dalton Hubble	209efd2f5b	Update Prometheus, Grafana, and kube-state-metrics	2021-09-29 23:39:10 -07:00
dependabot[bot]	388b1238bc	Bump mkdocs-material from 7.2.8 to 7.3.0 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.2.8 to 7.3.0. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.2.8...7.3.0) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2021-09-27 20:40:44 -07:00
Dalton Hubble	5a1e455220	Update nginx-ingress from v1.0.0 to v1.0.1	2021-09-24 09:38:18 -07:00
Dalton Hubble	69f37c8b17	Update Prometheus from v2.29.2 to v2.30.0	2021-09-24 09:34:00 -07:00
Dalton Hubble	b30de949b8	Update Calico and Cilium CNI * Update Calico from v3.20.0 to v3.20.1 * Update Cilium from v1.10.3 to v1.10.4	2021-09-22 22:18:16 -07:00
dependabot[bot]	4973178750	Bump mkdocs-material from 7.2.6 to 7.2.8 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.2.6 to 7.2.8. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.2.6...7.2.8) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-09-21 08:59:11 -07:00
Dalton Hubble	bb7f31822e	Update Kubernetes from v1.22.1 to v1.22.2 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222	2021-09-15 19:56:24 -07:00
Anthony Rabbito	c6923b9ef3	Switch Fedora CoreOS to new ARM64 AMIs (#1038 ) * Fedora CoreOS now publishes ARM64 AMIs	2021-09-12 11:49:13 -07:00
Dalton Hubble	dae79d5916	Remove mention of freenode IRC See #995	2021-09-12 10:10:49 -07:00
dependabot[bot]	f4d5ac0ca7	Bump mkdocs-material from 7.2.5 to 7.2.6 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.2.5 to 7.2.6. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.2.5...7.2.6) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-09-11 12:49:06 -07:00
Dalton Hubble	7e1b2cdba1	Discontinue Docker automated build publishing * Poseidon infra publishes official multi-arch container images for Kubelet to both Quay and Dockerhub (fallback). There is no change here * Automated builds by Quay and Dockerhub added separately tagged images for those not able to trust our images and preferring to trust Quay/Dockerhub. Going forward, we're ending the use of Dockerhub automated builds. Docker has moved automated builds to paid plans, even for open source projects (we're not petitioning for a special exemption given these are our unofficial images). Those still needing Kubelet images built externally (i.e. not Poseidon Labs) would still be able to use the Quay images tagged `build-SHA`	2021-09-01 11:52:57 -07:00
dependabot[bot]	3bb20ce083	Bump mkdocs-material from 7.2.4 to 7.2.5 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.2.4 to 7.2.5. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.2.4...7.2.5) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-08-31 17:34:24 -07:00
Dalton Hubble	eb29fb639b	Update nginx-ingress, Prometheus, and Grafana addons	2021-08-24 22:14:57 -07:00
Dalton Hubble	fcbdb50d93	Update Kubernetes from v1.22.0 to v1.22.1 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221	2021-08-19 21:12:02 -07:00
dependabot[bot]	efac611e9c	Bump mkdocs-material from 7.2.2 to 7.2.4 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.2.2 to 7.2.4. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.2.2...7.2.4) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-08-16 11:51:08 -07:00
dependabot[bot]	87ff431b80	Bump pygments from 2.9.0 to 2.10.0 Bumps [pygments](https://github.com/pygments/pygments) from 2.9.0 to 2.10.0. - [Release notes](https://github.com/pygments/pygments/releases) - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES) - [Commits](https://github.com/pygments/pygments/compare/2.9.0...2.10.0) --- updated-dependencies: - dependency-name: pygments dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2021-08-16 11:40:09 -07:00
Dalton Hubble	0d8ceae1d9	Add etcd v3.5.0 note to CHANGES	2021-08-11 09:24:43 -07:00