Add Prometheus config for monitoring Kubernetes Ingress

* Allow Kubernetes Ingress resources to be probed via Blackbox Exporter (if present) if annotated `prometheus.io/probe: "true"` * Fix probes of Services via Blackbox Exporter. Require Blackbox Exporter to be deployed in the same `monitoring` namespace, be named `blackbox-exporter`, and use port 8080
Configure Prometheus to allow a custom scrape query param
2025-08-02 17:51:33 +02:00 · 2021-12-29 11:57:50 -08:00 · 2021-12-29 11:47:10 -08:00 · 2021-12-28 13:25:23 -08:00 · 2021-12-23 14:33:26 -08:00 · 2021-12-20 08:36:19 -08:00
70 changed files with 295 additions and 214 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,21 @@ Notable changes between versions.

 ## Latest

+* Kubernetes [v1.23.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1231)
+* Workaround Terraform v1.1 regression in `file` provisioner ([#1093](https://github.com/poseidon/typhoon/pull/1093))
+
+### Flatcar Linux
+
+* Switch Kubernetes Container Runtime from `docker` to `containerd` ([#1087](https://github.com/poseidon/typhoon/pull/1087))
+
+### Addons
+
+* Configure Prometheus to allow a custom scrape query parameter ([#1095](https://github.com/poseidon/typhoon/pull/1095))
+* Configure Prometheus to probe Kubernetes Ingress via `blackbox-exporter` ([#1096](https://github.com/poseidon/typhoon/pull/1096))
+* Fix Prometheus Service probes to use `blackbox-exporter`, not `blackbox` ([#1096](https://github.com/poseidon/typhoon/pull/1096))
+
+## v1.23.0
+
 * Kubernetes [v1.23.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230)
 * Normalize CA cert mounts in static Pods and kube-proxy ([#1078](https://github.com/poseidon/typhoon/pull/1078))
 * Set Kubelet resolver config to `/run/systemd/resolve/resolv.conf` ([#1082](https://github.com/poseidon/typhoon/pull/1082))
--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@ -58,7 +58,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -97,9 +97,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.1
 ```

 List the pods.
--- a/addons/prometheus/config.yaml
+++ b/addons/prometheus/config.yaml
@ -175,6 +175,7 @@ data:
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
+    # * `prometheus.io/param`: Custom metrics query parameter, like "format=prometheus".
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
@ -197,6 +198,11 @@ data:
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_param]
+        action: replace
+        target_label: __param_$1
+        regex: ([^=]+)=(.*)
+        replacement: $2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
@ -214,38 +220,6 @@ data:
        action: drop
        regex: etcd_(debugging|disk|request|server).*

-    # Example scrape config for probing services via the Blackbox Exporter.
-    #
-    # The relabeling allows the actual service scrape endpoint to be configured
-    # via the following annotations:
-    #
-    # * `prometheus.io/probe`: Only probe services that have a value of `true`
-    - job_name: 'kubernetes-services'
-
-      metrics_path: /probe
-      params:
-        module: [http_2xx]
-
-      kubernetes_sd_configs:
-      - role: service
-
-      relabel_configs:
-      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
-        action: keep
-        regex: true
-      - source_labels: [__address__]
-        target_label: __param_target
-      - target_label: __address__
-        replacement: blackbox
-      - source_labels: [__param_target]
-        target_label: instance
-      - action: labelmap
-        regex: __meta_kubernetes_service_label_(.+)
-      - source_labels: [__meta_kubernetes_namespace]
-        target_label: namespace
-      - source_labels: [__meta_kubernetes_service_name]
-        target_label: job
-
    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
@ -282,6 +256,67 @@ data:
        action: replace
        target_label: kubernetes_pod_name

+    # Example scrape config for probing Services via the Blackbox Exporter.
+    #
+    # Relabeling allows service scraping to be configured via annotations:
+    # * `prometheus.io/probe`: Only probe services that have a value of `true`
+    - job_name: 'kubernetes-services'
+
+      metrics_path: /probe
+      params:
+        module: [http_2xx]
+
+      kubernetes_sd_configs:
+      - role: service
+
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
+        action: keep
+        regex: true
+      - source_labels: [__address__]
+        target_label: __param_target
+      - target_label: __address__
+        replacement: blackbox-exporter:8080
+      - source_labels: [__param_target]
+        target_label: instance
+      - action: labelmap
+        regex: __meta_kubernetes_service_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_service_name]
+        target_label: job
+
+    # Example scrape config for probing Ingresses via a Blackbox Exporter.
+    #
+    # Relabeling allows service scraping to be configured via annotations:
+    # * `prometheus.io/probe`: Only probe ingresses that have a value of `true`
+    - job_name: 'kubernetes-ingresses'
+      metrics_path: /probe
+      params:
+        module: [http_2xx]
+
+      kubernetes_sd_configs:
+      - role: ingress
+
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
+        regex: (.+);(.+);(.+)
+        replacement: ${1}://${2}${3}
+        target_label: __param_target
+      - target_label: __address__
+        replacement: blackbox-exporter:8080
+      - source_labels: [__param_target]
+        target_label: instance
+      - action: labelmap
+        regex: __meta_kubernetes_ingress_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_service_name]
+        target_label: job
+
    # Rule files
    rule_files:
      - "/etc/prometheus/rules/*.rules"
--- a/addons/prometheus/rbac/cluster-role.yaml
+++ b/addons/prometheus/rbac/cluster-role.yaml
@ -10,6 +10,17 @@ rules:
  - services
  - endpoints
  - pods
-  verbs: ["get", "list", "watch"]
+  verbs:
+  - get
+  - list
+  - watch
 - nonResourceURLs: ["/metrics"]
  verbs: ["get"]
+- apiGroups:
+  - networking.k8s.io
+  resources:
+  - ingresses
+  verbs:
+  - get
+  - list
+  - watch
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -54,7 +54,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -123,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.23.0
+            quay.io/poseidon/kubelet:v1.23.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/aws/fedora-coreos/kubernetes/ssh.tf
+++ b/aws/fedora-coreos/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -91,7 +91,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/aws/flatcar-linux/kubernetes/README.md
+++ b/aws/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/aws/flatcar-linux/kubernetes/bootstrap.tf
+++ b/aws/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/aws/flatcar-linux/kubernetes/cl/controller.yaml
@ -57,7 +57,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -78,7 +78,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -88,6 +88,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -120,7 +122,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/aws/flatcar-linux/kubernetes/ssh.tf
+++ b/aws/flatcar-linux/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -29,7 +29,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -53,7 +53,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -63,6 +63,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -95,7 +97,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/fcc/controller.yaml
@ -51,7 +51,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -118,7 +118,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.23.0
+            quay.io/poseidon/kubelet:v1.23.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/azure/fedora-coreos/kubernetes/ssh.tf
+++ b/azure/fedora-coreos/kubernetes/ssh.tf
@ -25,7 +25,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -24,7 +24,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -86,7 +86,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/azure/flatcar-linux/kubernetes/README.md
+++ b/azure/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/azure/flatcar-linux/kubernetes/bootstrap.tf
+++ b/azure/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/azure/flatcar-linux/kubernetes/cl/controller.yaml
@ -55,7 +55,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -75,7 +75,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -85,6 +85,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -116,7 +118,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/azure/flatcar-linux/kubernetes/ssh.tf
+++ b/azure/flatcar-linux/kubernetes/ssh.tf
@ -25,7 +25,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -50,7 +50,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -60,6 +60,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -91,7 +93,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -50,7 +50,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -120,7 +120,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -23,7 +23,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
--- a/bare-metal/fedora-coreos/kubernetes/ssh.tf
+++ b/bare-metal/fedora-coreos/kubernetes/ssh.tf
@ -28,17 +28,17 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo touch /etc/kubernetes",
      "sudo /opt/bootstrap/layout",
    ]
@ -65,12 +65,12 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo touch /etc/kubernetes",
    ]
  }
--- a/bare-metal/flatcar-linux/kubernetes/README.md
+++ b/bare-metal/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/controller.yaml
@ -63,7 +63,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -83,7 +83,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -93,6 +93,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -125,7 +127,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/flatcar-linux/kubernetes/cl/worker.yaml
@ -35,7 +35,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -58,7 +58,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -68,6 +68,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
--- a/bare-metal/flatcar-linux/kubernetes/ssh.tf
+++ b/bare-metal/flatcar-linux/kubernetes/ssh.tf
@ -29,17 +29,17 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo /opt/bootstrap/layout",
    ]
  }
@ -66,12 +66,12 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
    ]
  }
 }
--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -52,7 +52,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -130,7 +130,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.23.0
+            quay.io/poseidon/kubelet:v1.23.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -26,7 +26,7 @@ systemd:
        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/afterburn
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -93,7 +93,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/digital-ocean/fedora-coreos/kubernetes/ssh.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/ssh.tf
@ -25,17 +25,17 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo touch /etc/kubernetes",
      "sudo /opt/bootstrap/layout",
    ]
@ -55,12 +55,12 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo touch /etc/kubernetes",
    ]
  }
--- a/digital-ocean/flatcar-linux/kubernetes/README.md
+++ b/digital-ocean/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/controller.yaml
@ -65,7 +65,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -86,7 +86,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -96,6 +96,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -128,7 +130,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/flatcar-linux/kubernetes/cl/worker.yaml
@ -37,7 +37,7 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        EnvironmentFile=/run/metadata/coreos
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -61,7 +61,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -71,6 +71,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -97,7 +99,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/digital-ocean/flatcar-linux/kubernetes/ssh.tf
+++ b/digital-ocean/flatcar-linux/kubernetes/ssh.tf
@ -25,17 +25,17 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
      "sudo /opt/bootstrap/layout",
    ]
  }
@ -54,12 +54,12 @@ resource "null_resource" "copy-worker-secrets" {

  provisioner "file" {
    content     = module.bootstrap.kubeconfig-kubelet
-    destination = "$HOME/kubeconfig"
+    destination = "/home/core/kubeconfig"
  }

  provisioner "remote-exec" {
    inline = [
-      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
    ]
  }
 }
--- a/docs/advanced/arm64.md
+++ b/docs/advanced/arm64.md
@ -14,7 +14,7 @@ Create a cluster with ARM64 controller and worker nodes. Container workloads mus

 ```tf
 module "gravitas" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.1"

  # AWS
  cluster_name = "gravitas"
@ -40,9 +40,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes.
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-12-178   Ready    <none>   101s   v1.23.0   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-18-93    Ready    <none>   102s   v1.23.0   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
-ip-10-0-90-10    Ready    <none>   104s   v1.23.0   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-12-178   Ready    <none>   101s   v1.23.1   10.0.12.178   <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-18-93    Ready    <none>   102s   v1.23.1   10.0.18.93    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
+ip-10-0-90-10    Ready    <none>   104s   v1.23.1   10.0.90.10    <none>        Fedora CoreOS 32.20201104.dev.0   5.8.17-200.fc32.aarch64   docker://19.3.11
 ```

 ## Hybrid
@ -53,7 +53,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo

    ```tf
    module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.1"

      # AWS
      cluster_name = "gravitas"
@ -76,7 +76,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo

    ```tf
    module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.1"

      # AWS
      vpc_id          = module.gravitas.vpc_id
@ -101,9 +101,9 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 ```
 $ kubectl get nodes -o wide
 NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION             CONTAINER-RUNTIME
-ip-10-0-1-81    Ready    <none>   4m28s   v1.23.0   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-17-86   Ready    <none>   4m28s   v1.23.0   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
-ip-10-0-21-45   Ready    <none>   4m28s   v1.23.0   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
-ip-10-0-40-36   Ready    <none>   4m22s   v1.23.0   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-1-81    Ready    <none>   4m28s   v1.23.1   10.0.1.81     <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-17-86   Ready    <none>   4m28s   v1.23.1   10.0.17.86    <none>        Fedora CoreOS 33.20210413.dev.0   5.10.19-200.fc33.aarch64   docker://19.3.13
+ip-10-0-21-45   Ready    <none>   4m28s   v1.23.1   10.0.21.45    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
+ip-10-0-40-36   Ready    <none>   4m22s   v1.23.1   10.0.40.36    <none>        Fedora CoreOS 34.20210427.3.0     5.11.15-300.fc34.x86_64    docker://20.10.6
 ```

--- a/docs/advanced/nodes.md
+++ b/docs/advanced/nodes.md
@ -36,7 +36,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to

    ```tf
    module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.1"

      # Google Cloud
      cluster_name  = "yavin"
@ -57,7 +57,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to

    ```tf
    module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.1"

      # Google Cloud
      cluster_name = "yavin"
@ -89,7 +89,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and

    ```tf
    module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.1"

      # Google Cloud
      cluster_name  = "yavin"
@ -110,7 +110,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and

    ```tf
    module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.1"

      # Google Cloud
      cluster_name = "yavin"
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).

    ```tf
    module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.23.1"

      # AWS
      vpc_id          = module.tempest.vpc_id
@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).

    ```tf
    module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.23.1"

      # AWS
      vpc_id          = module.tempest.vpc_id
@ -111,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste

    ```tf
    module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.23.1"

      # Azure
      region                  = module.ramius.region
@ -137,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste

    ```tf
    module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.23.1"

      # Azure
      region                  = module.ramius.region
@ -207,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c

    ```tf
    module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.23.1"

      # Google Cloud
      region       = "europe-west2"
@ -231,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c

    ```tf
    module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.23.0"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.23.1"

      # Google Cloud
      region       = "europe-west2"
@ -262,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.23.0
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.23.0
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.23.0
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.23.0
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.23.0
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.23.1
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.23.1
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.23.1
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.23.1
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.23.1
 ```

 ### Variables
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.23.1"

  # AWS
  cluster_name = "tempest"
@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.23.0
-ip-10-0-26-65  Ready   <none>   10m  v1.23.0
-ip-10-0-41-21  Ready   <none>   10m  v1.23.0
+ip-10-0-3-155  Ready   <none>   10m  v1.23.1
+ip-10-0-26-65  Ready   <none>   10m  v1.23.1
+ip-10-0-41-21  Ready   <none>   10m  v1.23.1
 ```

 List the pods.
--- a/docs/fedora-coreos/azure.md
+++ b/docs/fedora-coreos/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on Azure with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -86,7 +86,7 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.23.1"

  # Azure
  cluster_name   = "ramius"
@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.23.0
-ramius-worker-000001  Ready   <none>  25m  v1.23.0
-ramius-worker-000002  Ready   <none>  24m  v1.23.0
+ramius-controller-0   Ready   <none>  24m  v1.23.1
+ramius-worker-000001  Ready   <none>  25m  v1.23.1
+ramius-worker-000002  Ready   <none>  24m  v1.23.1
 ```

 List the pods.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.1 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.23.1"

  # bare-metal
  cluster_name            = "mercury"
@ -283,9 +283,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.23.0
-node2.example.com   Ready   <none>  10m  v1.23.0
-node3.example.com   Ready   <none>  10m  v1.23.0
+node1.example.com   Ready   <none>  10m  v1.23.1
+node2.example.com   Ready   <none>  10m  v1.23.1
+node3.example.com   Ready   <none>  10m  v1.23.1
 ```

 List the pods.
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@ -1,6 +1,6 @@
 # DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on DigitalOcean with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.23.1"

  # Digital Ocean
  cluster_name = "nemo"
@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.23.0
-10.132.115.81      Ready   <none>  10m  v1.23.0
-10.132.124.107     Ready   <none>  10m  v1.23.0
+10.132.110.130     Ready   <none>  10m  v1.23.1
+10.132.115.81      Ready   <none>  10m  v1.23.1
+10.132.124.107     Ready   <none>  10m  v1.23.1
 ```

 List the pods.
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on Google Compute Engine with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.1
 ```

 List the pods.
--- a/docs/flatcar-linux/aws.md
+++ b/docs/flatcar-linux/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on AWS with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.23.1"

  # AWS
  cluster_name = "tempest"
@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.23.0
-ip-10-0-26-65  Ready   <none>  10m  v1.23.0
-ip-10-0-41-21  Ready   <none>  10m  v1.23.0
+ip-10-0-3-155  Ready   <none>  10m  v1.23.1
+ip-10-0-26-65  Ready   <none>  10m  v1.23.1
+ip-10-0-41-21  Ready   <none>  10m  v1.23.1
 ```

 List the pods.
--- a/docs/flatcar-linux/azure.md
+++ b/docs/flatcar-linux/azure.md
@ -1,6 +1,6 @@
 # Azure

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on Azure with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -75,7 +75,7 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.23.1"

  # Azure
  cluster_name   = "ramius"
@ -149,9 +149,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.23.0
-ramius-worker-000001  Ready   <none>  25m  v1.23.0
-ramius-worker-000002  Ready   <none>  24m  v1.23.0
+ramius-controller-0   Ready   <none>  24m  v1.23.1
+ramius-worker-000001  Ready   <none>  25m  v1.23.1
+ramius-worker-000002  Ready   <none>  24m  v1.23.1
 ```

 List the pods.
--- a/docs/flatcar-linux/bare-metal.md
+++ b/docs/flatcar-linux/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.23.0 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.23.1 cluster on bare-metal with Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.1"

  # bare-metal
  cluster_name            = "mercury"
@ -293,9 +293,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.23.0
-node2.example.com   Ready   <none>  10m  v1.23.0
-node3.example.com   Ready   <none>  10m  v1.23.0
+node1.example.com   Ready   <none>  10m  v1.23.1
+node2.example.com   Ready   <none>  10m  v1.23.1
+node3.example.com   Ready   <none>  10m  v1.23.1
 ```

 List the pods.
--- a/docs/flatcar-linux/digitalocean.md
+++ b/docs/flatcar-linux/digitalocean.md
@ -1,6 +1,6 @@
 # DigitalOcean

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on DigitalOcean with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.23.1"

  # Digital Ocean
  cluster_name = "nemo"
@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.23.0
-10.132.115.81      Ready   <none>  10m  v1.23.0
-10.132.124.107     Ready   <none>  10m  v1.23.0
+10.132.110.130     Ready   <none>  10m  v1.23.1
+10.132.115.81      Ready   <none>  10m  v1.23.1
+10.132.124.107     Ready   <none>  10m  v1.23.1
 ```

 List the pods.
--- a/docs/flatcar-linux/google-cloud.md
+++ b/docs/flatcar-linux/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.23.0 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.23.1 cluster on Google Compute Engine with Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -92,7 +92,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.23.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -167,9 +167,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.1
 ```

 List the pods.
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@ -57,7 +57,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -95,9 +95,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.23.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.23.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.23.1
 ```

 List the pods.
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar

 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.23.1"
  ...
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.23.1"
  ...
 }
 ```
--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
+++ b/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
@ -51,7 +51,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -118,7 +118,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.23.0
+            quay.io/poseidon/kubelet:v1.23.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/google-cloud/fedora-coreos/kubernetes/ssh.tf
+++ b/google-cloud/fedora-coreos/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -24,7 +24,7 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -86,7 +86,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/google-cloud/flatcar-linux/kubernetes/README.md
+++ b/google-cloud/flatcar-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.23.0 (upstream)
+* Kubernetes v1.23.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
--- a/google-cloud/flatcar-linux/kubernetes/bootstrap.tf
+++ b/google-cloud/flatcar-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=37f45cb28be2188befb5304794ba312cd8048fab"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4dc03881498ea715deff34925255f518f54d9513"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/flatcar-linux/kubernetes/cl/controller.yaml
+++ b/google-cloud/flatcar-linux/kubernetes/cl/controller.yaml
@ -55,7 +55,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -75,7 +75,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -85,6 +85,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -116,7 +118,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStart=/usr/bin/docker run \
            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
            -v /opt/bootstrap/assets:/assets:ro \
--- a/google-cloud/flatcar-linux/kubernetes/ssh.tf
+++ b/google-cloud/flatcar-linux/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/google-cloud/flatcar-linux/kubernetes/workers/cl/worker.yaml
+++ b/google-cloud/flatcar-linux/kubernetes/workers/cl/worker.yaml
@ -27,7 +27,7 @@ systemd:
        After=docker.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -50,7 +50,7 @@ systemd:
          -v /run:/run \
          -v /sys/fs/cgroup:/sys/fs/cgroup \
          -v /var/lib/calico:/var/lib/calico:ro \
-          -v /var/lib/docker:/var/lib/docker \
+          -v /var/lib/containerd:/var/lib/containerd \
          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
          -v /var/log:/var/log \
          -v /opt/cni/bin:/opt/cni/bin \
@ -60,6 +60,8 @@ systemd:
          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
          --cgroup-driver=systemd \
+          --container-runtime=remote \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -91,7 +93,7 @@ systemd:
        [Unit]
        Description=Delete Kubernetes node on shutdown
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.0
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.1
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
--- a/requirements.txt
+++ b/requirements.txt
@ -1,4 +1,4 @@
 mkdocs==1.2.3
-mkdocs-material==8.1.1
+mkdocs-material==8.1.3
 pygments==2.10.0
 pymdown-extensions==9.1
Author	SHA1	Message	Date
Dalton Hubble	50215e373b	Add Prometheus config for monitoring Kubernetes Ingress * Allow Kubernetes Ingress resources to be probed via Blackbox Exporter (if present) if annotated `prometheus.io/probe: "true"` * Fix probes of Services via Blackbox Exporter. Require Blackbox Exporter to be deployed in the same `monitoring` namespace, be named `blackbox-exporter`, and use port 8080	2021-12-29 11:57:50 -08:00
Dalton Hubble	a9f9c59b91	Configure Prometheus to allow a custom scrape query param * Set `prometheus.io/param` on a Kubernetes Service to scrape the service endpoints and pass a custom query parameter * For example, scrape Consul with `?format=prometheus` ```yaml kind: Service metadata: annotations: prometheus.io/scrape: 'true' prometheus.io/port: '8500' prometheus.io/path: /v1/agent/metrics prometheus.io/param: format=prometheus ```	2021-12-29 11:47:10 -08:00
Dalton Hubble	6ed048eb65	Workaround Terraform v1.1 file provisioner regression * Terraform v1.1 changed the behavior of provisioners and `remote-exec` in a way that breaks support for expansions in commands (including file provisioner, where `destination` is part of an `scp` command) * Terraform will likely revert the change eventually, but I suspect it will take a while * Instead, we can stop relying on Terraform's expansion behavior. `/home/core` is a suitable choice for `$HOME` on both Flatcar Linux and Fedora CoreOS (harldink `/var/home/core`) Rel: https://github.com/hashicorp/terraform/issues/30243	2021-12-28 13:25:23 -08:00
dependabot[bot]	ce7b2fa21f	Bump mkdocs-material from 8.1.1 to 8.1.3 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 8.1.1 to 8.1.3. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/8.1.1...8.1.3) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-12-23 14:33:26 -08:00
Dalton Hubble	9e3807798f	Update Kubernetes from v1.23.0 to v1.23.1 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1231	2021-12-20 08:36:19 -08:00
Dalton Hubble	ef9c6aa423	Switch Flatcar Linux to using containerd CRI * Use containerd as the Kubernetes Container Runtime	2021-12-15 08:42:13 -08:00