Update Prometheus and Grafana addons

* Change to `backend_address_pool_ids` list

CHANGES.md

@@ -4,9 +4,115 @@ Notable changes between versions.
 
 ## Latest
 
+* Kubernetes [v1.23.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230)
+* Normalize CA cert mounts in static Pods and kube-proxy ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+* Set Kubelet resolver config to `/run/systemd/resolve/resolv.conf` ([#1082](https://github.com/poseidon/typhoon/pull/1082))
+* Update Cilium from v1.10.5 to [v1.11.0](https://github.com/cilium/cilium/releases/tag/v1.11.0) ([#1083](https://github.com/poseidon/typhoon/pull/1083))
+* With Calico, add missing `caliconodestatuses` CRD ([#289](https://github.com/poseidon/terraform-render-bootstrap/pull/289))
+* Change `enable_aggregation` default to true ([#279](https://github.com/poseidon/terraform-render-bootstrap/pull/279))
+* Remove deprecated `--port` from `kube-scheduler` ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+
+### AWS
+
+* Change controller node default `disk_iops` to 3000 ([#1073](https://github.com/poseidon/typhoon/pull/1073))
+
+### Azure
+
+* Fix warning about deprecated `backend_address_pool_id` ([#1086](https://github.com/poseidon/typhoon/pull/1086))
+
+### Fedora CoreOS
+
+* Fix Fedora ARM64 workers to official Fedora CoreOS AMIs ([#1072](https://github.com/poseidon/typhoon/pull/1072))
+  * Should have been changed alongside controller AMIs in ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+  * Old Poseidon built ARM64 AMIs have been deleted
+
+### Addons
+
+* Update nginx-ingress from v1.0.5 to [v1.1.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.0)
+* Update Prometheus from v2.31.1 to [v2.32.0](https://github.com/prometheus/prometheus/releases/tag/v2.32.0)
+* Update kube-state-metrics from v2.2.4 to [v2.3.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.3.0)
+* Update node-exporter from v1.3.0 to [v1.3.1](https://github.com/prometheus/node_exporter/releases/tag/v1.3.1)
+* Update Grafana from v8.2.4 to [v8.3.3](https://github.com/grafana/grafana/releases/tag/v8.3.3)
+
+### Known Issues
+
+* Calico does not yet support Kubernetes v1.23.0, use `flannel` or `cilium` ([calico#5011](https://github.com/projectcalico/calico/issues/5011))
+
+## v1.22.4
+
+* Kubernetes [v1.22.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224)
+* Update CoreDNS from v1.8.4 to [v1.8.6](https://github.com/poseidon/terraform-render-bootstrap/pull/284)
+* Update Calico from v3.20.2 to [v3.21.0](https://github.com/projectcalico/calico/releases/tag/v3.21.0)
+* Update flannel from v0.14.0 to [v0.15.1](https://github.com/flannel-io/flannel/releases/tag/v0.15.1)
+
+### Google
+
+* Allow use of Terraform provider `google` [v4.0+](https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0)
+
+### Flatcar Linux
+
+* Change Kubelet mounts for cgroups v2 ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+* Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+
+### Addons
+
+* Update Prometheus from v2.30.3 to [v2.31.1](https://github.com/prometheus/prometheus/releases/tag/v2.31.1)
+* Update node-exporter from v1.2.2 to [v1.3.0](https://github.com/prometheus/node_exporter/releases/tag/v1.3.0)
+* Update kube-state-metrics from v2.2.3 to [v2.2.4](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.4)
+* Update Grafana from v8.2.1 to [v8.2.4](https://github.com/grafana/grafana/releases/tag/v8.2.4)
+* Update nginx-ingress from v1.0.4 to [v1.0.5](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.5)
+
+## v1.23.3
+
+* Kubernetes [v1.22.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223)
+* Update etcd from v3.5.0 to [v3.5.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.1)
+* Update Cilium from v1.10.4 to [v1.10.5](https://github.com/cilium/cilium/releases/tag/v1.10.5)
+* Update Calico from v3.20.1 to [v3.20.2](https://github.com/projectcalico/calico/releases/tag/v3.20.2)
+  * Use Calico's iptables legacy vs nft auto-detection
+* Update flannel from v0.13.0 to v0.14.0
+
+### Bare-Metal
+
+* Require Terraform provider `poseidon/matchbox` v0.5+ ([#1048](https://github.com/poseidon/typhoon/pull/1048))
+
+### Addons
+
+* Update nginx-ingress from v1.0.0 to [v1.0.4](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4)
+* Update Prometheus from v2.29.2 to [v2.30.3](https://github.com/prometheus/prometheus/releases/tag/v2.30.3)
+* Update kube-state-metrics from v2.2.0 to [v2.2.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.3)
+* Update Grafana from v8.1.2 to [v8.2.1](https://github.com/grafana/grafana/releases/tag/v8.2.1)
+
+## v1.22.2
+
+* Kubernetes [v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222)
+* Update Cilium from v1.10.3 to [v1.10.4](https://github.com/cilium/cilium/releases/tag/v1.10.4)
+* Update Calico from v3.20.0 to [v3.20.1](https://github.com/projectcalico/calico/releases/tag/v3.20.1)
+* Fix access to ClusterIP services with Cilium ([#276](https://github.com/poseidon/terraform-render-bootstrap/pull/276))
+
+### Fedora CoreOS
+
+* Use Fedora CoreOS ARM64 AMIs ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+
+### Addons
+
+* Update Prometheus from v2.29.1 to [v2.29.2](https://github.com/prometheus/prometheus/releases/tag/v2.29.2)
+* Update kube-state-metrics from v2.1.1 to [v2.2.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.0)
+
+## v1.22.1
+
+* Kubernetes [v1.22.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221)
+* Update Calico from v3.19.1 to [v3.20.0](https://github.com/projectcalico/calico/releases/tag/v3.20.0)
+
+### Addons
+
+* Update nginx-ingress from v1.0.0-beta.1 to [v1.0.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0)
+* Update Prometheus from v2.28.1 to [v2.29.1](https://github.com/prometheus/prometheus/releases/tag/v2.29.1)
+* Update Grafana from v8.1.1 to [v8.1.2](https://github.com/grafana/grafana/releases/tag/v8.1.2)
+
 ## v1.22.0
 
 * Kubernetes [v1.22.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1220)
+* Update etcd from v3.4.16 to [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0)
 * Switch `kube-controller-manager` and `kube-scheduler` to use secure port only
   * Update Prometheus config to discover endpoints and use a bearer token to scrape
 

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@@ -58,7 +58,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -97,9 +97,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```
 
 List the pods.

addons/grafana/deployment.yaml

@@ -24,7 +24,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: grafana
-          image: docker.io/grafana/grafana:8.1.1
+          image: docker.io/grafana/grafana:8.3.3
           env:
             - name: GF_PATHS_CONFIG
               value: "/etc/grafana/custom.ini"

addons/nginx-ingress/aws/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/azure/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: nginx-ingress-controller
-          image: k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.1
+          image: k8s.gcr.io/ingress-nginx/controller:v1.1.0
           args:
             - /nginx-ingress-controller
             - --ingress-class=public

addons/prometheus/deployment.yaml

@@ -21,7 +21,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.28.1
+          image: quay.io/prometheus/prometheus:v2.32.0
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -25,7 +25,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.1
+        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
         ports:
           - name: metrics
             containerPort: 8080

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -28,7 +28,7 @@ spec:
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.2.2
+        image: quay.io/prometheus/node-exporter:v1.3.1
         args:
           - --path.procfs=/host/proc
           - --path.sysfs=/host/sys

aws/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/fedora-coreos/kubernetes/ami.tf

@@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
   most_recent = true
   owners      = ["125523088429"]
@@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
   }
 }
 
-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
   count = var.arch == "arm64" ? 1 : 0
 
   most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]
 
   filter {
     name   = "architecture"
@@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
   }
 
   filter {
-    name   = "name"
+    name   = "description"
-    values = ["fedora-coreos-*"]
+    values = ["Fedora CoreOS ${var.os_stream} *"]
   }
 }
-

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -13,7 +13,5 @@ module "bootstrap" {
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }
 

aws/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -54,7 +54,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -97,6 +97,7 @@ systemd:
           --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -122,7 +123,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -218,6 +219,7 @@ storage:
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
           ETCD_UNSUPPORTED_ARCH=arm64
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

aws/fedora-coreos/kubernetes/variables.tf

@@ -66,8 +66,8 @@ variable "disk_type" {
 
 variable "disk_iops" {
   type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
+  description = "IOPS of the EBS volume (e.g. 3000)"
-  default     = 0
+  default     = 3000
 }
 
 variable "worker_price" {
@@ -142,8 +142,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 variable "worker_node_labels" {

aws/fedora-coreos/kubernetes/versions.tf

@@ -4,8 +4,8 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

aws/fedora-coreos/kubernetes/workers/ami.tf

@@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
   most_recent = true
   owners      = ["125523088429"]
@@ -19,14 +18,11 @@ data "aws_ami" "fedora-coreos" {
   }
 }
 
-# Experimental Fedora CoreOS arm64 / aarch64 AMIs from Poseidon
-# WARNING: These AMIs will be removed when Fedora CoreOS publishes arm64 AMIs
-# and may be removed for any reason before then as well. Do not use.
 data "aws_ami" "fedora-coreos-arm" {
   count = var.arch == "arm64" ? 1 : 0
 
   most_recent = true
-  owners      = ["099663496933"]
+  owners      = ["125523088429"]
 
   filter {
     name   = "architecture"
@@ -39,8 +35,7 @@ data "aws_ami" "fedora-coreos-arm" {
   }
 
   filter {
-    name   = "name"
+    name   = "description"
-    values = ["fedora-coreos-*"]
+    values = ["Fedora CoreOS ${var.os_stream} *"]
   }
 }
-

aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -27,7 +27,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -76,6 +76,7 @@ systemd:
           --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
@@ -90,7 +91,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -129,6 +130,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

aws/fedora-coreos/kubernetes/workers/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
 
     ct = {
       source  = "poseidon/ct"

aws/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -57,7 +57,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -76,8 +76,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -88,6 +87,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -98,6 +98,7 @@ systemd:
           --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -119,7 +120,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

aws/flatcar-linux/kubernetes/variables.tf

@@ -66,8 +66,8 @@ variable "disk_type" {
 
 variable "disk_iops" {
   type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
+  description = "IOPS of the EBS volume (e.g. 3000)"
-  default     = 0
+  default     = 3000
 }
 
 variable "worker_price" {
@@ -142,8 +142,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 variable "worker_node_labels" {

aws/flatcar-linux/kubernetes/versions.tf

@@ -4,8 +4,8 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

aws/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -29,7 +29,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -51,8 +51,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -63,6 +62,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -79,6 +79,7 @@ systemd:
           --pod-manifest-path=/etc/kubernetes/manifests \
           --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
@@ -94,7 +95,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

aws/flatcar-linux/kubernetes/workers/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     aws      = ">= 2.23, <= 4.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
 
     ct = {
       source  = "poseidon/ct"

azure/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -19,8 +19,5 @@ module "bootstrap" {
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }
 

azure/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -51,7 +51,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -92,6 +92,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -117,7 +118,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -212,6 +213,7 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

azure/fedora-coreos/kubernetes/lb.tf

@@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
   loadbalancer_id                = azurerm_lb.cluster.id
   frontend_ip_configuration_name = "apiserver"
 
-  protocol                = "Tcp"
+  protocol                 = "Tcp"
-  frontend_port           = 6443
+  frontend_port            = 6443
-  backend_port            = 6443
+  backend_port             = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
-  probe_id                = azurerm_lb_probe.apiserver.id
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }
 
 resource "azurerm_lb_rule" "ingress-http" {
@@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
   frontend_ip_configuration_name = "ingress"
   disable_outbound_snat          = true
 
-  protocol                = "Tcp"
+  protocol                 = "Tcp"
-  frontend_port           = 80
+  frontend_port            = 80
-  backend_port            = 80
+  backend_port             = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
-  probe_id                = azurerm_lb_probe.ingress.id
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
 resource "azurerm_lb_rule" "ingress-https" {
@@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
   frontend_ip_configuration_name = "ingress"
   disable_outbound_snat          = true
 
-  protocol                = "Tcp"
+  protocol                 = "Tcp"
-  frontend_port           = 443
+  frontend_port            = 443
-  backend_port            = 443
+  backend_port             = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
-  probe_id                = azurerm_lb_probe.ingress.id
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
 # Worker outbound TCP/UDP SNAT

azure/fedora-coreos/kubernetes/variables.tf

@@ -117,8 +117,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 variable "worker_node_labels" {

azure/fedora-coreos/kubernetes/versions.tf

@@ -4,8 +4,8 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -24,7 +24,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -71,6 +71,7 @@ systemd:
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
@@ -85,7 +86,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -124,6 +125,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

azure/fedora-coreos/kubernetes/workers/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"
 
     ct = {
       source  = "poseidon/ct"

azure/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

azure/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -55,7 +55,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -73,8 +73,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -85,6 +84,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -94,6 +94,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -115,7 +116,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

azure/flatcar-linux/kubernetes/lb.tf

@@ -59,11 +59,11 @@ resource "azurerm_lb_rule" "apiserver" {
   loadbalancer_id                = azurerm_lb.cluster.id
   frontend_ip_configuration_name = "apiserver"
 
-  protocol                = "Tcp"
+  protocol                 = "Tcp"
-  frontend_port           = 6443
+  frontend_port            = 6443
-  backend_port            = 6443
+  backend_port             = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
-  probe_id                = azurerm_lb_probe.apiserver.id
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }
 
 resource "azurerm_lb_rule" "ingress-http" {
@@ -74,11 +74,11 @@ resource "azurerm_lb_rule" "ingress-http" {
   frontend_ip_configuration_name = "ingress"
   disable_outbound_snat          = true
 
-  protocol                = "Tcp"
+  protocol                 = "Tcp"
-  frontend_port           = 80
+  frontend_port            = 80
-  backend_port            = 80
+  backend_port             = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
-  probe_id                = azurerm_lb_probe.ingress.id
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
 resource "azurerm_lb_rule" "ingress-https" {
@@ -89,11 +89,11 @@ resource "azurerm_lb_rule" "ingress-https" {
   frontend_ip_configuration_name = "ingress"
   disable_outbound_snat          = true
 
-  protocol                = "Tcp"
+  protocol                 = "Tcp"
-  frontend_port           = 443
+  frontend_port            = 443
-  backend_port            = 443
+  backend_port             = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
-  probe_id                = azurerm_lb_probe.ingress.id
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
 # Worker outbound TCP/UDP SNAT

azure/flatcar-linux/kubernetes/variables.tf

@@ -123,8 +123,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 variable "worker_node_labels" {

azure/flatcar-linux/kubernetes/versions.tf

@@ -4,8 +4,8 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

azure/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -27,7 +27,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -48,8 +48,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -60,6 +59,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -75,6 +75,7 @@ systemd:
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
@@ -90,7 +91,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

azure/flatcar-linux/kubernetes/workers/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
     azurerm  = "~> 2.8"
-    template = "~> 2.1"
+    template = "~> 2.2"
 
     ct = {
       source  = "poseidon/ct"

bare-metal/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]
@@ -13,8 +13,6 @@ module "bootstrap" {
   cluster_domain_suffix           = var.cluster_domain_suffix
   enable_reporting                = var.enable_reporting
   enable_aggregation              = var.enable_aggregation
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }
 
 

bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -50,7 +50,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -92,6 +92,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -119,7 +120,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
@@ -222,6 +223,7 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml

@@ -23,7 +23,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -71,6 +71,7 @@ systemd:
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
@@ -120,6 +121,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

bare-metal/fedora-coreos/kubernetes/profiles.tf

@@ -44,7 +44,7 @@ resource "matchbox_profile" "controllers" {
 
   kernel = local.kernel
   initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)
 
   raw_ignition = data.ct_config.controller-ignitions.*.rendered[count.index]
 }
@@ -78,7 +78,7 @@ resource "matchbox_profile" "workers" {
 
   kernel = local.kernel
   initrd = local.initrd
-  args = concat(local.args, var.kernel_args)
+  args   = concat(local.args, var.kernel_args)
 
   raw_ignition = data.ct_config.worker-ignitions.*.rendered[count.index]
 }

bare-metal/fedora-coreos/kubernetes/variables.tf

@@ -146,8 +146,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 # unofficial, undocumented, unsupported

bare-metal/fedora-coreos/kubernetes/versions.tf

@@ -3,8 +3,8 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"
@@ -13,7 +13,7 @@ terraform {
 
     matchbox = {
       source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
     }
   }
 }

bare-metal/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -63,7 +63,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -81,8 +81,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -93,6 +92,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -103,6 +103,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -124,7 +125,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

bare-metal/flatcar-linux/kubernetes/cl/worker.yaml

@@ -35,7 +35,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -56,8 +56,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -68,6 +67,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -84,6 +84,7 @@ systemd:
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet

bare-metal/flatcar-linux/kubernetes/variables.tf

@@ -151,8 +151,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 # unofficial, undocumented, unsupported

bare-metal/flatcar-linux/kubernetes/versions.tf

@@ -3,8 +3,8 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"
@@ -13,7 +13,7 @@ terraform {
 
     matchbox = {
       source  = "poseidon/matchbox"
-      version = "~> 0.4.1"
+      version = "~> 0.5.0"
     }
   }
 }

digital-ocean/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -17,8 +17,5 @@ module "bootstrap" {
   cluster_domain_suffix = var.cluster_domain_suffix
   enable_reporting      = var.enable_reporting
   enable_aggregation    = var.enable_aggregation
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
 }
 

digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -52,7 +52,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -95,6 +95,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -129,7 +130,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -219,3 +220,5 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
+

digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml

@@ -26,7 +26,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -69,6 +69,7 @@ systemd:
           --node-labels=node.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
@@ -92,7 +93,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -126,3 +127,4 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp

digital-ocean/fedora-coreos/kubernetes/variables.tf

@@ -94,8 +94,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 # unofficial, undocumented, unsupported

digital-ocean/fedora-coreos/kubernetes/versions.tf

@@ -3,8 +3,8 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

digital-ocean/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -65,7 +65,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -84,8 +84,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -96,6 +95,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -106,6 +106,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -127,7 +128,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml

@@ -37,7 +37,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -59,8 +59,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -71,6 +70,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -81,6 +81,7 @@ systemd:
           --node-labels=node.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
@@ -96,7 +97,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

digital-ocean/flatcar-linux/kubernetes/variables.tf

@@ -94,8 +94,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 # unofficial, undocumented, unsupported

digital-ocean/flatcar-linux/kubernetes/versions.tf

@@ -3,8 +3,8 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

docs/advanced/arm64.md

@@ -8,20 +8,13 @@ Typhoon has experimental support for ARM64 with Fedora CoreOS on AWS. Full clust
 !!! note
     Currently, CNI networking must be set to flannel or Cilium.
 
-## AMIs
-
-In lieu of official Fedora CoreOS ARM64 AMIs, Poseidon publishes experimental ARM64 AMIs to a few regions (us-east-1, us-east-2, us-west-1). These AMIs may be **removed** at any time and will be replaced when Fedora CoreOS publishes equivalents.
-
-!!! note
-    AMIs are only published to a few regions, and AWS availability of ARM instance types varies.
-
 ## Cluster
 
 Create a cluster with ARM64 controller and worker nodes. Container workloads must be `arm64` compatible and use `arm64` container images.
 
 ```tf
 module "gravitas" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # AWS
   cluster_name = "gravitas"
@@ -47,9 +40,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes.
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-12-178   Ready    <none>   101s   v1.22.0   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-12-178   Ready    <none>   101s   v1.23.0   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-18-93    Ready    <none>   102s   v1.22.0   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-18-93    Ready    <none>   102s   v1.23.0   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-90-10    Ready    <none>   104s   v1.22.0   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-90-10    Ready    <none>   104s   v1.23.0   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
 ```
 
 ## Hybrid
@@ -60,7 +53,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"
 
       # AWS
       cluster_name = "gravitas"
@@ -83,7 +76,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"
 
       # AWS
       vpc_id          = module.gravitas.vpc_id
@@ -108,9 +101,9 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 ```
 $ kubectl get nodes -o wide
 NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION             CONTAINER-RUNTIME
-ip-10-0-1-81    Ready    <none>   4m28s   v1.22.0   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-1-81    Ready    <none>   4m28s   v1.23.0   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-17-86   Ready    <none>   4m28s   v1.22.0   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
+ip-10-0-17-86   Ready    <none>   4m28s   v1.23.0   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
-ip-10-0-21-45   Ready    <none>   4m28s   v1.22.0   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-21-45   Ready    <none>   4m28s   v1.23.0   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-40-36   Ready    <none>   4m22s   v1.22.0   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-40-36   Ready    <none>   4m22s   v1.23.0   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
 ```
 

docs/advanced/nodes.md

@@ -36,7 +36,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to
 
     ```tf
     module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
 
       # Google Cloud
       cluster_name  = "yavin"
@@ -57,7 +57,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to
 
     ```tf
     module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"
 
       # Google Cloud
       cluster_name = "yavin"
@@ -89,7 +89,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and
 
     ```tf
     module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
 
       # Google Cloud
       cluster_name  = "yavin"
@@ -110,7 +110,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and
 
     ```tf
     module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"
 
       # Google Cloud
       cluster_name = "yavin"

docs/advanced/worker-pools.md

@@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.23.0"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -111,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.23.0"
 
       # Azure
       region                  = module.ramius.region
@@ -137,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.23.0"
 
       # Azure
       region                  = module.ramius.region
@@ -207,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"
 
       # Google Cloud
       region       = "europe-west2"
@@ -231,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.22.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.23.0"
 
       # Google Cloud
       region       = "europe-west2"
@@ -262,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.22.0
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.23.0
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.22.0
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.23.0
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.22.0
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.23.0
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.22.0
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.23.0
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.22.0
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.23.0
 ```
 
 ### Variables

docs/architecture/operating-systems.md

@@ -15,24 +15,26 @@ Together, they diversify Typhoon to support a range of container technologies.
 ## Host Properties
 
 | Property          | Flatcar Linux | Fedora CoreOS |
-|-------------------|---------------------------------|---------------|
+|-------------------|---------------|---------------|
-| Kernel            | ~5.10.x | ~5.12.x |
+| Kernel            | ~5.10.x       | ~5.14.x       |
-| systemd           | 247 | 248 |
+| systemd           | 247           | 249           |
+| Username          | core          | core          |
 | Ignition system   | Ignition v2.x spec | Ignition v3.x spec |
-| Container Engine  | docker 19.3.15  | docker 20.10.6 |
 | storage driver    | overlay2 (extfs)  | overlay2 (xfs) |
-| logging driver    | json-file | journald |
+| logging driver    | json-file     | journald      |
-| cgroup driver     | cgroupfs (except Flatcar edge) | systemd  |
+| cgroup driver     | systemd       | systemd       |
-| Networking        | systemd-networkd | NetworkManager |
+| cgroup version    | v2            | v2            |
-| Username          | core      | core |
+| Networking        | systemd-networkd | NetworkManager   |
+| Resolver          | systemd-resolved | systemd-resolved |
 
 ## Kubernetes Properties
 
 | Property          | Flatcar Linux | Fedora CoreOS |
-|-------------------|-----------------|---------------|
+|-------------------|---------------|---------------|
 | single-master     | all platforms | all platforms |
 | multi-master      | all platforms | all platforms |
 | control plane     | static pods   | static pods   |
+| Container Runtime | docker 20.10  | docker 20.10  |
 | kubelet image     | kubelet [image](https://github.com/poseidon/kubelet) with upstream binary | kubelet [image](https://github.com/poseidon/kubelet) with upstream binary |
 | control plane images | upstream images | upstream images |
 | on-host etcd      | docker    | podman |

docs/fedora-coreos/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -51,11 +51,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     aws = {
       source = "hashicorp/aws"
-      version = "3.48.0"
+      version = "3.67.0"
     }
   }
 }
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.22.0
+ip-10-0-3-155  Ready   <none>   10m  v1.23.0
-ip-10-0-26-65  Ready   <none>   10m  v1.22.0
+ip-10-0-26-65  Ready   <none>   10m  v1.23.0
-ip-10-0-41-21  Ready   <none>   10m  v1.22.0
+ip-10-0-41-21  Ready   <none>   10m  v1.23.0
 ```
 
 List the pods.

docs/fedora-coreos/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -48,11 +48,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     azurerm = {
       source = "hashicorp/azurerm"
-      version = "2.68.0"
+      version = "2.88.1"
     }
   }
 }
@@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # Azure
   cluster_name   = "ramius"
@@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.22.0
+ramius-controller-0   Ready   <none>  24m  v1.23.0
-ramius-worker-000001  Ready   <none>  25m  v1.22.0
+ramius-worker-000001  Ready   <none>  25m  v1.23.0
-ramius-worker-000002  Ready   <none>  24m  v1.22.0
+ramius-worker-000002  Ready   <none>  24m  v1.23.0
 ```
 
 List the pods.

docs/fedora-coreos/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.22.0 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Fedora CoreOS.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -138,11 +138,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     matchbox = {
       source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
     }
   }
 }
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -283,9 +283,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.22.0
+node1.example.com   Ready   <none>  10m  v1.23.0
-node2.example.com   Ready   <none>  10m  v1.22.0
+node2.example.com   Ready   <none>  10m  v1.23.0
-node3.example.com   Ready   <none>  10m  v1.22.0
+node3.example.com   Ready   <none>  10m  v1.23.0
 ```
 
 List the pods.

docs/fedora-coreos/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -51,7 +51,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     digitalocean = {
       source = "digitalocean/digitalocean"
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.22.0
+10.132.110.130     Ready   <none>  10m  v1.23.0
-10.132.115.81      Ready   <none>  10m  v1.22.0
+10.132.115.81      Ready   <none>  10m  v1.23.0
-10.132.124.107     Ready   <none>  10m  v1.22.0
+10.132.124.107     Ready   <none>  10m  v1.23.0
 ```
 
 List the pods.

docs/fedora-coreos/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -52,11 +52,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     google = {
       source = "hashicorp/google"
-      version = "3.75.0"
+      version = "4.3.0"
     }
   }
 }
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```
 
 List the pods.

docs/flatcar-linux/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -51,11 +51,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     aws = {
       source = "hashicorp/aws"
-      version = "3.48.0"
+      version = "3.67.0"
     }
   }
 }
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.0"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.22.0
+ip-10-0-3-155  Ready   <none>  10m  v1.23.0
-ip-10-0-26-65  Ready   <none>  10m  v1.22.0
+ip-10-0-26-65  Ready   <none>  10m  v1.23.0
-ip-10-0-41-21  Ready   <none>  10m  v1.22.0
+ip-10-0-41-21  Ready   <none>  10m  v1.23.0
 ```
 
 List the pods.

docs/flatcar-linux/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -48,11 +48,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     azurerm = {
       source = "hashicorp/azurerm"
-      version = "2.68.0"
+      version = "2.88.1"
     }
   }
 }
@@ -75,7 +75,7 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.23.0"
 
   # Azure
   cluster_name   = "ramius"
@@ -149,9 +149,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.22.0
+ramius-controller-0   Ready   <none>  24m  v1.23.0
-ramius-worker-000001  Ready   <none>  25m  v1.22.0
+ramius-worker-000001  Ready   <none>  25m  v1.23.0
-ramius-worker-000002  Ready   <none>  24m  v1.22.0
+ramius-worker-000002  Ready   <none>  24m  v1.23.0
 ```
 
 List the pods.

docs/flatcar-linux/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.22.0 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Flatcar Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -138,11 +138,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     matchbox = {
       source = "poseidon/matchbox"
-      version = "0.4.1"
+      version = "0.5.0"
     }
   }
 }
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -293,9 +293,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.22.0
+node1.example.com   Ready   <none>  10m  v1.23.0
-node2.example.com   Ready   <none>  10m  v1.22.0
+node2.example.com   Ready   <none>  10m  v1.23.0
-node3.example.com   Ready   <none>  10m  v1.22.0
+node3.example.com   Ready   <none>  10m  v1.23.0
 ```
 
 List the pods.

docs/flatcar-linux/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -51,7 +51,7 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     digitalocean = {
       source = "digitalocean/digitalocean"
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.23.0"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.22.0
+10.132.110.130     Ready   <none>  10m  v1.23.0
-10.132.115.81      Ready   <none>  10m  v1.22.0
+10.132.115.81      Ready   <none>  10m  v1.23.0
-10.132.124.107     Ready   <none>  10m  v1.22.0
+10.132.124.107     Ready   <none>  10m  v1.23.0
 ```
 
 List the pods.

docs/flatcar-linux/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.22.0 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -52,11 +52,11 @@ terraform {
   required_providers {
     ct = {
       source  = "poseidon/ct"
-      version = "0.9.0"
+      version = "0.9.1"
     }
     google = {
       source = "hashicorp/google"
-      version = "3.75.0"
+      version = "4.3.0"
     }
   }
 }
@@ -92,7 +92,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.23.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -167,9 +167,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```
 
 List the pods.

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@@ -57,7 +57,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -95,9 +95,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.22.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.22.0
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
 ```
 
 List the pods.

docs/topics/faq.md

@@ -6,10 +6,6 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 
 Formats rise and evolve. Typhoon may choose to adapt the format over time (with lots of forewarning). However, the authors' have built several Kubernetes "distros" before and learned from mistakes - Terraform modules are the right format for now.
 
-## Get Help
-
-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
-
 ## Security Issues
 
 If you find security issues, please see [security disclosures](/topics/security/#disclosures).

docs/topics/maintenance.md

@@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar
 
 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
   ...
 }
 
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.22.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"
   ...
 }
 ```

docs/topics/security.md

@@ -62,9 +62,9 @@ The Kubelet image is published to Quay.io and Dockerhub.
 Two tag styles indicate the build strategy used.
 
 * Typhoon internal infra publishes single and multi-arch images (e.g. `v1.18.4`, `v1.18.4-amd64`, `v1.18.4-arm64`, `v1.18.4-2-g23228e6-amd64`, `v1.18.4-2-g23228e6-arm64`)
-* Quay and Dockerhub automated builds publish verifiable images (e.g. `build-SHA` on Quay, `build-TAG` on Dockerhub)
+* Quay automated builds publish verifiable images (e.g. `build-SHA` on Quay)
 
-The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay/Dockerhub (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization/#system-images).
+The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization/#system-images).
 
 ### flannel-cni
 

google-cloud/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

google-cloud/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -14,8 +14,6 @@ module "bootstrap" {
   enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
 
-  trusted_certs_dir = "/etc/pki/tls/certs"
-
   // temporary
   external_apiserver_port = 443
 }

google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml

@@ -12,7 +12,7 @@ systemd:
         Wants=network-online.target network.target
         After=network-online.target
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         Type=exec
         ExecStartPre=/bin/mkdir -p /var/lib/etcd
         ExecStartPre=-/usr/bin/podman rm etcd
@@ -51,7 +51,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -92,6 +92,7 @@ systemd:
           --node-labels=node.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -117,7 +118,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.22.0
+            quay.io/poseidon/kubelet:v1.23.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -212,6 +213,7 @@ storage:
           ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
           ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
           ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

google-cloud/fedora-coreos/kubernetes/variables.tf

@@ -118,8 +118,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 variable "worker_node_labels" {

google-cloud/fedora-coreos/kubernetes/versions.tf

@@ -3,9 +3,9 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    google   = ">= 2.19, < 4.0"
+    google   = ">= 2.19, < 5.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml

@@ -24,7 +24,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -71,6 +71,7 @@ systemd:
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/podman stop kubelet
@@ -85,7 +86,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true
@@ -124,6 +125,7 @@ storage:
           DefaultCPUAccounting=yes
           DefaultMemoryAccounting=yes
           DefaultBlockIOAccounting=yes
+    - path: /etc/fedora-coreos/iptables-legacy.stamp
 passwd:
   users:
     - name: core

google-cloud/fedora-coreos/kubernetes/workers/versions.tf

@@ -3,8 +3,8 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    google   = ">= 2.19, < 4.0"
+    google   = ">= 2.19, < 5.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
 
     ct = {
       source  = "poseidon/ct"

google-cloud/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.22.0 (upstream)
+* Kubernetes v1.23.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

google-cloud/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=b5f5d843ec9babcd2eeea98b8edcef972a5c178d"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

google-cloud/flatcar-linux/kubernetes/cl/controller.yaml

@@ -10,7 +10,7 @@ systemd:
         Requires=docker.service
         After=docker.service
         [Service]
-        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.0
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.1
         ExecStartPre=/usr/bin/docker run -d \
           --name etcd \
           --network host \
@@ -55,7 +55,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -73,8 +73,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -85,6 +84,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -95,6 +95,7 @@ systemd:
           --pod-manifest-path=/etc/kubernetes/manifests \
           --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
@@ -115,7 +116,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

google-cloud/flatcar-linux/kubernetes/variables.tf

@@ -112,8 +112,8 @@ variable "enable_reporting" {
 
 variable "enable_aggregation" {
   type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  description = "Enable the Kubernetes Aggregation Layer"
-  default     = false
+  default     = true
 }
 
 variable "worker_node_labels" {

google-cloud/flatcar-linux/kubernetes/versions.tf

@@ -3,9 +3,9 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    google   = ">= 2.19, < 4.0"
+    google   = ">= 2.19, < 5.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
-    null     = "~> 2.1"
+    null     = ">= 2.1"
 
     ct = {
       source  = "poseidon/ct"

google-cloud/flatcar-linux/kubernetes/workers/cl/worker.yaml

@@ -27,7 +27,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -48,8 +48,7 @@ systemd:
           -v /usr/lib/os-release:/etc/os-release:ro \
           -v /lib/modules:/lib/modules:ro \
           -v /run:/run \
-          -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
-          -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
           -v /var/lib/calico:/var/lib/calico:ro \
           -v /var/lib/docker:/var/lib/docker \
           -v /var/lib/kubelet:/var/lib/kubelet:rshared \
@@ -60,6 +59,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --cgroup-driver=systemd \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -75,6 +75,7 @@ systemd:
           %{~ endfor ~}
           --pod-manifest-path=/etc/kubernetes/manifests \
           --read-only-port=0 \
+          --resolv-conf=/run/systemd/resolve/resolv.conf \
           --rotate-certificates \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStart=docker logs -f kubelet
@@ -90,7 +91,7 @@ systemd:
         [Unit]
         Description=Delete Kubernetes node on shutdown
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.22.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
         Type=oneshot
         RemainAfterExit=true
         ExecStart=/bin/true

google-cloud/flatcar-linux/kubernetes/workers/versions.tf

@@ -3,8 +3,8 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    google   = ">= 2.19, < 4.0"
+    google   = ">= 2.19, < 5.0"
-    template = "~> 2.1"
+    template = "~> 2.2"
 
     ct = {
       source  = "poseidon/ct"