Adding install_snippets support.

During the "real" first boot (install boot), we need tu run butane
config to manipulate disks, so we add install_snippets variable to do
so.

This snippets are added to the install.yaml butane configuration

CHANGES.md

@@ -4,6 +4,101 @@ Notable changes between versions.
 
 ## Latest
 
+### Azure
+
+* Allow controller and worker nodes to use different CPU architectures
+  * Add `controller_arch` and `worker_arch` variables
+  * Remove the `arch` variable
+
+## v1.30.3
+
+* Kubernetes [v1.30.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1303)
+* Update Cilium from v1.15.6 to [v1.15.7](https://github.com/cilium/cilium/releases/tag/v1.15.7)
+* Update flannel from v0.25.4 to [v0.25.5](https://github.com/flannel-io/flannel/releases/tag/v0.25.5)
+
+### AWS
+
+* Allow configuring controller and worker disks ([#1482](https://github.com/poseidon/typhoon/pull/1482))
+  * Add `controller_disk_type`, `controller_disk_size`, and `controller_disk_iops` variables
+  * Add `worker_disk_type`, `worker_disk_size`, and `worker_disk_iops` variables
+  * Remove `disk_type`, `disk_size`, and `disk_iops` variables
+  * Fix propagating settings to worker disks, previously ignored
+* Allow configuring CPU pricing model for burstable instance types ([#1482](https://github.com/poseidon/typhoon/pull/1482))
+  * Add `controller_cpu_credits` and `worker_cpu_credits` variables (`standard` or `unlimited`)
+* Configure controller or worker instance architecture ([#1485](https://github.com/poseidon/typhoon/pull/1485))
+  * Add `controller_arch` and `worker_arch` variables (`amd64` or `arm64`)
+  * Remove `arch` variable
+
+```diff
+module "cluster" {
+  ...
+- arch      = "amd64"
+- disk_type = "gp3"
+- disk_size = 30
+- disk_iops = 3000
+
++ controller_arch        = "amd64"
++ controller_disk_size   = 15
++ controller_cpu_credits = "standard"
++ worker_arch            = "amd64"
++ worker_disk_size       = 22
++ worker_cpu_credits     = "unlimited"
+}
+```
+
+### Azure
+
+* Configure the virtual network and subnets with IPv6 private address space
+  * Change `host_cidr` variable (string) to a `network_cidr` object with `ipv4` and `ipv6` fields that list CIDR strings. Leave the variable unset to use the defaults. (**breaking**)
+* Add support for dual-stack Kubernetes Ingress Load Balancing
+  * Add a public IPv6 frontend, 80/443 rules, and a worker-ipv6 backend pool
+  * Change the `controller_address_prefixes` output from a list of strings to an object with `ipv4` and `ipv6` fields. Most Azure resources can't accept a mix, so these are split out (**breaking**)
+  * Change the `worker_address_prefixes` output from a list of strings to an object with `ipv4` and `ipv6` fields. Most Azure resources can't accept a mix, so these are split out (**breaking**)
+  * Change the `backend_address_pool_id` output (and worker module input) from a string to an object with `ipv4` and `ipv6` fields that list ids (**breaking**)
+* Configure nodes to have outbound IPv6 internet connectivity (analogous to IPv4 SNAT)
+  * Configure controller nodes to have a public IPv6 address
+  * Configure worker nodes to use outbound rules and the load balancer for SNAT
+* Extend network security rules to allow IPv6 traffic, analogous to IPv4
+* Rename `region` variable to `location` to align with Azure platform conventions ([#1469](https://github.com/poseidon/typhoon/pull/1469))
+* Change worker pools from uniform to flexible orchestration mode ([#1473](https://github.com/poseidon/typhoon/pull/1473))
+* Add options to allow workers nodes to use ephemeral local disks ([#1473](https://github.com/poseidon/typhoon/pull/1473))
+  * Add `controller_disk_type` and `controller_disk_size` variables
+  * Add `worker_disk_type`, `worker_disk_size`, and `worker_ephemeral_disk` variables
+* Reduce the number of public IPv4 addresses needed for the Azure load balancer ([#1470](https://github.com/poseidon/typhoon/pull/1470))
+* Configure controller or worker instance architecture for Flatcar Linux ([#1485](https://github.com/poseidon/typhoon/pull/1485))
+  * Add `controller_arch` and `worker_arch` variables (`amd64` or `arm64`)
+  * Remove `arch` variable
+
+```diff
+module "cluster" {
+  ...
+- region = "centralus"
++ location = "centralus"
+  # optional
+- host_cidr = "10.0.0.0/16"
++ network_cidr = {
++   ipv4 = ["10.0.0.0/16"]
++ }
+
+  # instances
++ controller_disk_type = "StandardSSD_LRS"
++ worker_ephemeral_disk = true
+}
+```
+
+### Google Cloud
+
+* Allow configuring controller and worker disks ([#1486](https://github.com/poseidon/typhoon/pull/1486))
+  * Add `controller_disk_size` and `worker_disk_size` variables
+  * Remove `disk_size` variable
+
+## v1.30.2
+
+* Kubernetes [v1.30.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1302)
+* Update CoreDNS from v1.9.4 to v1.11.1
+* Update Cilium from v1.15.5 to [v1.15.6](https://github.com/cilium/cilium/releases/tag/v1.15.6)
+* Update flannel from v0.25.1 to [v0.25.4](https://github.com/flannel-io/flannel/releases/tag/v0.25.4)
+
 ## v1.30.1
 
 * Kubernetes [v1.30.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1301)

README.md

@@ -18,7 +18,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@@ -78,7 +78,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -117,9 +117,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
 ```
 
 List the pods.

addons/cilium/daemonset.tf

@@ -61,7 +61,7 @@ resource "kubernetes_daemonset" "cilium" {
         # https://github.com/cilium/cilium/pull/24075
         init_container {
           name    = "install-cni"
-          image   = "quay.io/cilium/cilium:v1.15.5"
+          image   = "quay.io/cilium/cilium:v1.16.0"
           command = ["/install-plugin.sh"]
           security_context {
             allow_privilege_escalation = true
@@ -80,7 +80,7 @@ resource "kubernetes_daemonset" "cilium" {
         # We use nsenter command with host's cgroup and mount namespaces enabled.
         init_container {
           name  = "mount-cgroup"
-          image = "quay.io/cilium/cilium:v1.15.5"
+          image = "quay.io/cilium/cilium:v1.16.0"
           command = [
             "sh",
             "-ec",
@@ -115,7 +115,7 @@ resource "kubernetes_daemonset" "cilium" {
 
         init_container {
           name    = "clean-cilium-state"
-          image   = "quay.io/cilium/cilium:v1.15.5"
+          image   = "quay.io/cilium/cilium:v1.16.0"
           command = ["/init-container.sh"]
           security_context {
             allow_privilege_escalation = true
@@ -139,7 +139,7 @@ resource "kubernetes_daemonset" "cilium" {
 
         container {
           name    = "cilium-agent"
-          image   = "quay.io/cilium/cilium:v1.15.5"
+          image   = "quay.io/cilium/cilium:v1.16.0"
           command = ["cilium-agent"]
           args = [
             "--config-dir=/tmp/cilium/config-map"

addons/cilium/deployment.tf

@@ -58,7 +58,7 @@ resource "kubernetes_deployment" "operator" {
         enable_service_links            = false
         container {
           name    = "cilium-operator"
-          image   = "quay.io/cilium/operator-generic:v1.15.5"
+          image   = "quay.io/cilium/operator-generic:v1.16.0"
           command = ["cilium-operator-generic"]
           args = [
             "--config-dir=/tmp/cilium/config-map",

addons/coredns/deployment.tf

@@ -77,7 +77,7 @@ resource "kubernetes_deployment" "coredns" {
         }
         container {
           name  = "coredns"
-          image = "registry.k8s.io/coredns/coredns:v1.11.1"
+          image = "registry.k8s.io/coredns/coredns:v1.11.3"
           args  = ["-conf", "/etc/coredns/Corefile"]
           port {
             name           = "dns"

addons/flannel/daemonset.tf

@@ -73,7 +73,7 @@ resource "kubernetes_daemonset" "flannel" {
 
         container {
           name  = "flannel"
-          image = "docker.io/flannel/flannel:v0.25.1"
+          image = "docker.io/flannel/flannel:v0.25.5"
           command = [
             "/opt/bin/flanneld",
             "--ip-masq",

aws/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/fedora-coreos/kubernetes/ami.tf

@@ -19,7 +19,7 @@ data "aws_ami" "fedora-coreos" {
 }
 
 data "aws_ami" "fedora-coreos-arm" {
-  count = var.arch == "arm64" ? 1 : 0
+  count = var.controller_arch == "arm64" ? 1 : 0
 
   most_recent = true
   owners      = ["125523088429"]

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/fedora-coreos/kubernetes/butane/controller.yaml

@@ -57,7 +57,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -116,7 +116,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.1
+            quay.io/poseidon/kubelet:v1.30.3
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

aws/fedora-coreos/kubernetes/controllers.tf

@@ -22,14 +22,14 @@ resource "aws_instance" "controllers" {
   }
 
   instance_type = var.controller_type
-  ami           = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+  ami           = var.controller_arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
   user_data     = data.ct_config.controllers.*.rendered[count.index]
 
   # storage
   root_block_device {
-    volume_type = var.disk_type
-    volume_size = var.disk_size
-    iops        = var.disk_iops
+    volume_type = var.controller_disk_type
+    volume_size = var.controller_disk_size
+    iops        = var.controller_disk_iops
     encrypted   = true
     tags        = {}
   }
@@ -39,6 +39,10 @@ resource "aws_instance" "controllers" {
   subnet_id                   = element(aws_subnet.public.*.id, count.index)
   vpc_security_group_ids      = [aws_security_group.controller.id]
 
+  credit_specification {
+    cpu_credits = var.controller_cpu_credits
+  }
+
   lifecycle {
     ignore_changes = [
       ami,

aws/fedora-coreos/kubernetes/variables.tf

@@ -17,30 +17,6 @@ variable "dns_zone_id" {
 
 # instances
 
-variable "controller_count" {
-  type        = number
-  description = "Number of controllers (i.e. masters)"
-  default     = 1
-}
-
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
-variable "controller_type" {
-  type        = string
-  description = "EC2 instance type for controllers"
-  default     = "t3.small"
-}
-
-variable "worker_type" {
-  type        = string
-  description = "EC2 instance type for workers"
-  default     = "t3.small"
-}
-
 variable "os_stream" {
   type        = string
   description = "Fedora CoreOS image stream for instances (e.g. stable, testing, next)"
@@ -52,24 +28,78 @@ variable "os_stream" {
   }
 }
 
-variable "disk_size" {
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "EC2 instance type for controllers"
+  default     = "t3.small"
+}
+
+variable "controller_disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
   default     = 30
 }
 
-variable "disk_type" {
+variable "controller_disk_type" {
   type        = string
   description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
   default     = "gp3"
 }
 
-variable "disk_iops" {
+variable "controller_disk_iops" {
   type        = number
   description = "IOPS of the EBS volume (e.g. 3000)"
   default     = 3000
 }
 
+variable "controller_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "worker_type" {
+  type        = string
+  description = "EC2 instance type for workers"
+  default     = "t3.small"
+}
+
+variable "worker_disk_size" {
+  type        = number
+  description = "Size of the EBS volume in GB"
+  default     = 30
+}
+
+variable "worker_disk_type" {
+  type        = string
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
+}
+
+variable "worker_disk_iops" {
+  type        = number
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
+}
+
+variable "worker_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
 variable "worker_price" {
   type        = number
   description = "Spot price in USD for worker instances or 0 to use on-demand instances"
@@ -160,14 +190,25 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
-variable "arch" {
-  type        = string
-  description = "Container architecture (amd64 or arm64)"
-  default     = "amd64"
+# advanced
 
+variable "controller_arch" {
+  type        = string
+  description = "Controller node(s) architecture (amd64 or arm64)"
+  default     = "amd64"
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
-    error_message = "The arch must be amd64 or arm64."
+    condition     = contains(["amd64", "arm64"], var.controller_arch)
+    error_message = "The controller_arch must be amd64 or arm64."
+  }
+}
+
+variable "worker_arch" {
+  type        = string
+  description = "Worker node(s) architecture (amd64 or arm64)"
+  default     = "amd64"
+  validation {
+    condition     = contains(["amd64", "arm64"], var.worker_arch)
+    error_message = "The worker_arch must be amd64 or arm64."
   }
 }
 

aws/fedora-coreos/kubernetes/workers.tf

@@ -9,8 +9,11 @@ module "workers" {
   worker_count    = var.worker_count
   instance_type   = var.worker_type
   os_stream       = var.os_stream
-  arch            = var.arch
-  disk_size       = var.disk_size
+  arch            = var.worker_arch
+  disk_type       = var.worker_disk_type
+  disk_size       = var.worker_disk_size
+  disk_iops       = var.worker_disk_iops
+  cpu_credits     = var.worker_cpu_credits
   spot_price      = var.worker_price
   target_groups   = var.worker_target_groups
 

aws/fedora-coreos/kubernetes/workers/butane/worker.yaml

@@ -29,7 +29,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests

aws/fedora-coreos/kubernetes/workers/variables.tf

@@ -69,6 +69,12 @@ variable "spot_price" {
   default     = 0
 }
 
+variable "cpu_credits" {
+  type        = string
+  description = "CPU burst credits mode (if applicable)"
+  default     = null
+}
+
 variable "target_groups" {
   type        = list(string)
   description = "Additional target group ARNs to which instances should be added"

aws/fedora-coreos/kubernetes/workers/workers.tf

@@ -94,6 +94,10 @@ resource "aws_launch_template" "worker" {
     }
   }
 
+  credit_specification {
+    cpu_credits = var.cpu_credits
+  }
+
   lifecycle {
     // Override the default destroy and replace update behavior
     create_before_destroy = true

aws/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/flatcar-linux/kubernetes/ami.tf

@@ -1,7 +1,7 @@
 locals {
   # Pick a Flatcar Linux AMI
   # flatcar-stable -> Flatcar Linux AMI
-  ami_id  = var.arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
+  ami_id  = var.controller_arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
   channel = split("-", var.os_image)[1]
 }
 
@@ -26,7 +26,7 @@ data "aws_ami" "flatcar" {
 }
 
 data "aws_ami" "flatcar-arm64" {
-  count = var.arch == "arm64" ? 1 : 0
+  count = var.controller_arch == "arm64" ? 1 : 0
 
   most_recent = true
   owners      = ["075585003325"]

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

aws/flatcar-linux/kubernetes/butane/controller.yaml

@@ -58,7 +58,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -109,7 +109,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

aws/flatcar-linux/kubernetes/controllers.tf

@@ -28,9 +28,9 @@ resource "aws_instance" "controllers" {
 
   # storage
   root_block_device {
-    volume_type = var.disk_type
-    volume_size = var.disk_size
-    iops        = var.disk_iops
+    volume_type = var.controller_disk_type
+    volume_size = var.controller_disk_size
+    iops        = var.controller_disk_iops
     encrypted   = true
     tags        = {}
   }
@@ -40,6 +40,10 @@ resource "aws_instance" "controllers" {
   subnet_id                   = element(aws_subnet.public.*.id, count.index)
   vpc_security_group_ids      = [aws_security_group.controller.id]
 
+  credit_specification {
+    cpu_credits = var.controller_cpu_credits
+  }
+
   lifecycle {
     ignore_changes = [
       ami,

aws/flatcar-linux/kubernetes/variables.tf

@@ -17,30 +17,6 @@ variable "dns_zone_id" {
 
 # instances
 
-variable "controller_count" {
-  type        = number
-  description = "Number of controllers (i.e. masters)"
-  default     = 1
-}
-
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
-variable "controller_type" {
-  type        = string
-  description = "EC2 instance type for controllers"
-  default     = "t3.small"
-}
-
-variable "worker_type" {
-  type        = string
-  description = "EC2 instance type for workers"
-  default     = "t3.small"
-}
-
 variable "os_image" {
   type        = string
   description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
@@ -52,24 +28,78 @@ variable "os_image" {
   }
 }
 
-variable "disk_size" {
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "EC2 instance type for controllers"
+  default     = "t3.small"
+}
+
+variable "controller_disk_size" {
   type        = number
   description = "Size of the EBS volume in GB"
   default     = 30
 }
 
-variable "disk_type" {
+variable "controller_disk_type" {
   type        = string
   description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
   default     = "gp3"
 }
 
-variable "disk_iops" {
+variable "controller_disk_iops" {
   type        = number
   description = "IOPS of the EBS volume (e.g. 3000)"
   default     = 3000
 }
 
+variable "controller_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "worker_type" {
+  type        = string
+  description = "EC2 instance type for workers"
+  default     = "t3.small"
+}
+
+variable "worker_disk_size" {
+  type        = number
+  description = "Size of the EBS volume in GB"
+  default     = 30
+}
+
+variable "worker_disk_type" {
+  type        = string
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
+}
+
+variable "worker_disk_iops" {
+  type        = number
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
+}
+
+variable "worker_cpu_credits" {
+  type        = string
+  description = "CPU credits mode (if using a burstable instance type)"
+  default     = null
+}
+
 variable "worker_price" {
   type        = number
   description = "Spot price in USD for worker instances or 0 to use on-demand instances"
@@ -160,17 +190,27 @@ variable "cluster_domain_suffix" {
   default     = "cluster.local"
 }
 
-variable "arch" {
+variable "controller_arch" {
   type        = string
-  description = "Container architecture (amd64 or arm64)"
+  description = "Controller node(s) architecture (amd64 or arm64)"
   default     = "amd64"
-
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
-    error_message = "The arch must be amd64 or arm64."
+    condition     = contains(["amd64", "arm64"], var.controller_arch)
+    error_message = "The controller_arch must be amd64 or arm64."
   }
 }
 
+variable "worker_arch" {
+  type        = string
+  description = "Worker node(s) architecture (amd64 or arm64)"
+  default     = "amd64"
+  validation {
+    condition     = contains(["amd64", "arm64"], var.worker_arch)
+    error_message = "The worker_arch must be amd64 or arm64."
+  }
+}
+
+
 variable "daemonset_tolerations" {
   type        = list(string)
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"

aws/flatcar-linux/kubernetes/workers.tf

@@ -9,8 +9,10 @@ module "workers" {
   worker_count    = var.worker_count
   instance_type   = var.worker_type
   os_image        = var.os_image
-  arch            = var.arch
-  disk_size       = var.disk_size
+  arch            = var.worker_arch
+  disk_type       = var.worker_disk_type
+  disk_size       = var.worker_disk_size
+  disk_iops       = var.worker_disk_iops
   spot_price      = var.worker_price
   target_groups   = var.worker_target_groups
 

aws/flatcar-linux/kubernetes/workers/butane/worker.yaml

@@ -30,7 +30,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests

aws/flatcar-linux/kubernetes/workers/variables.tf

@@ -69,6 +69,12 @@ variable "spot_price" {
   default     = 0
 }
 
+variable "cpu_credits" {
+  type        = string
+  description = "CPU burst credits mode (if applicable)"
+  default     = null
+}
+
 variable "target_groups" {
   type        = list(string)
   description = "Additional target group ARNs to which instances should be added"

aws/flatcar-linux/kubernetes/workers/workers.tf

@@ -94,6 +94,10 @@ resource "aws_launch_template" "worker" {
     }
   }
 
+  credit_specification {
+    cpu_credits = var.cpu_credits
+  }
+
   lifecycle {
     // Override the default destroy and replace update behavior
     create_before_destroy = true

azure/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

azure/fedora-coreos/kubernetes/butane/controller.yaml

@@ -54,7 +54,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -111,7 +111,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.1
+            quay.io/poseidon/kubelet:v1.30.3
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

azure/fedora-coreos/kubernetes/controllers.tf

@@ -8,26 +8,23 @@ locals {
 
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "azurerm_dns_a_record" "etcds" {
-  count               = var.controller_count
-  resource_group_name = var.dns_zone_group
+  count = var.controller_count
 
   # DNS Zone name where record should be created
-  zone_name = var.dns_zone
-
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
   # DNS record
   name = format("%s-etcd%d", var.cluster_name, count.index)
   ttl  = 300
-
   # private IPv4 address for etcd
-  records = [azurerm_network_interface.controllers.*.private_ip_address[count.index]]
+  records = [azurerm_network_interface.controllers[count.index].private_ip_address]
 }
 
 # Controller availability set to spread controllers
 resource "azurerm_availability_set" "controllers" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                         = "${var.cluster_name}-controllers"
-  location                     = var.region
+  resource_group_name          = azurerm_resource_group.cluster.name
+  location                     = var.location
   platform_fault_domain_count  = 2
   platform_update_domain_count = 4
   managed                      = true
@@ -35,31 +32,35 @@ resource "azurerm_availability_set" "controllers" {
 
 # Controller instances
 resource "azurerm_linux_virtual_machine" "controllers" {
-  count               = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
+  count = var.controller_count
 
   name                = "${var.cluster_name}-controller-${count.index}"
-  location            = var.region
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
   availability_set_id = azurerm_availability_set.controllers.id
-
-  size        = var.controller_type
-  custom_data = base64encode(data.ct_config.controllers.*.rendered[count.index])
+  size                = var.controller_type
 
   # storage
   source_image_id = var.os_image
   os_disk {
     name                 = "${var.cluster_name}-controller-${count.index}"
+    storage_account_type = var.controller_disk_type
+    disk_size_gb         = var.controller_disk_size
     caching              = "None"
-    disk_size_gb         = var.disk_size
-    storage_account_type = "Premium_LRS"
   }
 
   # network
   network_interface_ids = [
-    azurerm_network_interface.controllers.*.id[count.index]
+    azurerm_network_interface.controllers[count.index].id
   ]
 
-  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
+  # boot
+  custom_data = base64encode(data.ct_config.controllers[count.index].rendered)
+  boot_diagnostics {
+    # defaults to a managed storage account
+  }
+
+  # Azure requires an RSA admin_ssh_key
   admin_username = "core"
   admin_ssh_key {
     username   = "core"
@@ -74,31 +75,52 @@ resource "azurerm_linux_virtual_machine" "controllers" {
   }
 }
 
-# Controller public IPv4 addresses
-resource "azurerm_public_ip" "controllers" {
-  count               = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
+# Controller node public IPv4 addresses
+resource "azurerm_public_ip" "controllers-ipv4" {
+  count = var.controller_count
 
-  name              = "${var.cluster_name}-controller-${count.index}"
-  location          = azurerm_resource_group.cluster.location
-  sku               = "Standard"
-  allocation_method = "Static"
+  name                = "${var.cluster_name}-controller-${count.index}-ipv4"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
+  ip_version          = "IPv4"
+  sku                 = "Standard"
+  allocation_method   = "Static"
 }
 
-# Controller NICs with public and private IPv4
-resource "azurerm_network_interface" "controllers" {
-  count               = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
+# Controller node public IPv6 addresses
+resource "azurerm_public_ip" "controllers-ipv6" {
+  count = var.controller_count
 
-  name     = "${var.cluster_name}-controller-${count.index}"
-  location = azurerm_resource_group.cluster.location
+  name                = "${var.cluster_name}-controller-${count.index}-ipv6"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
+  ip_version          = "IPv6"
+  sku                 = "Standard"
+  allocation_method   = "Static"
+}
+
+# Controllers' network interfaces
+resource "azurerm_network_interface" "controllers" {
+  count = var.controller_count
+
+  name                = "${var.cluster_name}-controller-${count.index}"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
 
   ip_configuration {
-    name                          = "ip0"
+    name                          = "ipv4"
+    primary                       = true
     subnet_id                     = azurerm_subnet.controller.id
     private_ip_address_allocation = "Dynamic"
-    # instance public IPv4
-    public_ip_address_id = azurerm_public_ip.controllers.*.id[count.index]
+    private_ip_address_version    = "IPv4"
+    public_ip_address_id          = azurerm_public_ip.controllers-ipv4[count.index].id
+  }
+  ip_configuration {
+    name                          = "ipv6"
+    subnet_id                     = azurerm_subnet.controller.id
+    private_ip_address_allocation = "Dynamic"
+    private_ip_address_version    = "IPv6"
+    public_ip_address_id          = azurerm_public_ip.controllers-ipv6[count.index].id
   }
 }
 
@@ -111,12 +133,20 @@ resource "azurerm_network_interface_security_group_association" "controllers" {
 }
 
 # Associate controller network interface with controller backend address pool
-resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+resource "azurerm_network_interface_backend_address_pool_association" "controllers-ipv4" {
   count = var.controller_count
 
   network_interface_id    = azurerm_network_interface.controllers[count.index].id
-  ip_configuration_name   = "ip0"
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  ip_configuration_name   = "ipv4"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller-ipv4.id
+}
+
+resource "azurerm_network_interface_backend_address_pool_association" "controllers-ipv6" {
+  count = var.controller_count
+
+  network_interface_id    = azurerm_network_interface.controllers[count.index].id
+  ip_configuration_name   = "ipv6"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller-ipv6.id
 }
 
 # Fedora CoreOS controllers

azure/fedora-coreos/kubernetes/lb.tf

@@ -1,116 +1,164 @@
-# DNS record for the apiserver load balancer
+# DNS A record for the apiserver load balancer
 resource "azurerm_dns_a_record" "apiserver" {
-  resource_group_name = var.dns_zone_group
-
   # DNS Zone name where record should be created
-  zone_name = var.dns_zone
-
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
   # DNS record
   name = var.cluster_name
   ttl  = 300
-
   # IPv4 address of apiserver load balancer
-  records = [azurerm_public_ip.apiserver-ipv4.ip_address]
+  records = [azurerm_public_ip.frontend-ipv4.ip_address]
 }
 
-# Static IPv4 address for the apiserver frontend
-resource "azurerm_public_ip" "apiserver-ipv4" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name              = "${var.cluster_name}-apiserver-ipv4"
-  location          = var.region
-  sku               = "Standard"
-  allocation_method = "Static"
+# DNS AAAA record for the apiserver load balancer
+resource "azurerm_dns_aaaa_record" "apiserver" {
+  # DNS Zone name where record should be created
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
+  # DNS record
+  name = var.cluster_name
+  ttl  = 300
+  # IPv4 address of apiserver load balancer
+  records = [azurerm_public_ip.frontend-ipv6.ip_address]
 }
 
-# Static IPv4 address for the ingress frontend
-resource "azurerm_public_ip" "ingress-ipv4" {
+# Static IPv4 address for the load balancer
+resource "azurerm_public_ip" "frontend-ipv4" {
+  name                = "${var.cluster_name}-frontend-ipv4"
   resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
+  ip_version          = "IPv4"
+  sku                 = "Standard"
+  allocation_method   = "Static"
+}
 
-  name              = "${var.cluster_name}-ingress-ipv4"
-  location          = var.region
-  sku               = "Standard"
-  allocation_method = "Static"
+# Static IPv6 address for the load balancer
+resource "azurerm_public_ip" "frontend-ipv6" {
+  name                = "${var.cluster_name}-frontend-ipv6"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
+  ip_version          = "IPv6"
+  sku                 = "Standard"
+  allocation_method   = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress
 resource "azurerm_lb" "cluster" {
+  name                = var.cluster_name
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = var.cluster_name
-  location = var.region
-  sku      = "Standard"
+  location            = var.location
+  sku                 = "Standard"
 
   frontend_ip_configuration {
-    name                 = "apiserver"
-    public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
+    name                 = "frontend-ipv4"
+    public_ip_address_id = azurerm_public_ip.frontend-ipv4.id
   }
 
   frontend_ip_configuration {
-    name                 = "ingress"
-    public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
+    name                 = "frontend-ipv6"
+    public_ip_address_id = azurerm_public_ip.frontend-ipv6.id
   }
 }
 
-resource "azurerm_lb_rule" "apiserver" {
-  name                           = "apiserver"
+resource "azurerm_lb_rule" "apiserver-ipv4" {
+  name                           = "apiserver-ipv4"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "apiserver"
+  frontend_ip_configuration_name = "frontend-ipv4"
+  disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 6443
   backend_port             = 6443
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller-ipv4.id]
   probe_id                 = azurerm_lb_probe.apiserver.id
 }
 
-resource "azurerm_lb_rule" "ingress-http" {
-  name                           = "ingress-http"
+resource "azurerm_lb_rule" "apiserver-ipv6" {
+  name                           = "apiserver-ipv6"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller-ipv6.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
+}
+
+resource "azurerm_lb_rule" "ingress-http-ipv4" {
+  name                           = "ingress-http-ipv4"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv4"
   disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 80
   backend_port             = 80
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv4.id]
   probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-resource "azurerm_lb_rule" "ingress-https" {
-  name                           = "ingress-https"
+resource "azurerm_lb_rule" "ingress-https-ipv4" {
+  name                           = "ingress-https-ipv4"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "frontend-ipv4"
   disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 443
   backend_port             = 443
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv4.id]
   probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-# Worker outbound TCP/UDP SNAT
-resource "azurerm_lb_outbound_rule" "worker-outbound" {
-  name            = "worker"
-  loadbalancer_id = azurerm_lb.cluster.id
-  frontend_ip_configuration {
-    name = "ingress"
-  }
+resource "azurerm_lb_rule" "ingress-http-ipv6" {
+  name                           = "ingress-http-ipv6"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
 
-  protocol                = "All"
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
+resource "azurerm_lb_rule" "ingress-https-ipv6" {
+  name                           = "ingress-https-ipv6"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
+}
+
+# Backend Address Pools
+
 # Address pool of controllers
-resource "azurerm_lb_backend_address_pool" "controller" {
-  name            = "controller"
+resource "azurerm_lb_backend_address_pool" "controller-ipv4" {
+  name            = "controller-ipv4"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+resource "azurerm_lb_backend_address_pool" "controller-ipv6" {
+  name            = "controller-ipv6"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
 # Address pool of workers
-resource "azurerm_lb_backend_address_pool" "worker" {
-  name            = "worker"
+resource "azurerm_lb_backend_address_pool" "worker-ipv4" {
+  name            = "worker-ipv4"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+resource "azurerm_lb_backend_address_pool" "worker-ipv6" {
+  name            = "worker-ipv6"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
@@ -122,10 +170,8 @@ resource "azurerm_lb_probe" "apiserver" {
   loadbalancer_id = azurerm_lb.cluster.id
   protocol        = "Tcp"
   port            = 6443
-
   # unhealthy threshold
-  number_of_probes = 3
-
+  number_of_probes    = 3
   interval_in_seconds = 5
 }
 
@@ -136,10 +182,29 @@ resource "azurerm_lb_probe" "ingress" {
   protocol        = "Http"
   port            = 10254
   request_path    = "/healthz"
-
   # unhealthy threshold
-  number_of_probes = 3
-
+  number_of_probes    = 3
   interval_in_seconds = 5
 }
 
+# Outbound SNAT
+
+resource "azurerm_lb_outbound_rule" "outbound-ipv4" {
+  name                    = "outbound-ipv4"
+  protocol                = "All"
+  loadbalancer_id         = azurerm_lb.cluster.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker-ipv4.id
+  frontend_ip_configuration {
+    name = "frontend-ipv4"
+  }
+}
+
+resource "azurerm_lb_outbound_rule" "outbound-ipv6" {
+  name                    = "outbound-ipv6"
+  protocol                = "All"
+  loadbalancer_id         = azurerm_lb.cluster.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker-ipv6.id
+  frontend_ip_configuration {
+    name = "frontend-ipv6"
+  }
+}

azure/fedora-coreos/kubernetes/locals.tf

@@ -0,0 +1,6 @@
+locals {
+  backend_address_pool_ids = {
+    ipv4 = [azurerm_lb_backend_address_pool.worker-ipv4.id]
+    ipv6 = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  }
+}

azure/fedora-coreos/kubernetes/network.tf

@@ -1,27 +1,64 @@
+# Choose an IPv6 ULA subnet at random
+# https://datatracker.ietf.org/doc/html/rfc4193
+resource "random_id" "ula-netnum" {
+  byte_length = 5 # 40 bits
+}
+
+locals {
+  # fd00::/8 -> shift 40 -> 2^40 possible /48 subnets
+  ula-range = cidrsubnet("fd00::/8", 40, random_id.ula-netnum.dec)
+  network_cidr = {
+    ipv4 = var.network_cidr.ipv4
+    ipv6 = length(var.network_cidr.ipv6) > 0 ? var.network_cidr.ipv6 : [local.ula-range]
+  }
+
+  # Subdivide the virtual network into subnets
+  # - controllers use netnum 0
+  # - workers use netnum 1
+  controller_subnets = {
+    ipv4 = [for i, cidr in local.network_cidr.ipv4 : cidrsubnet(cidr, 1, 0)]
+    ipv6 = [for i, cidr in local.network_cidr.ipv6 : cidrsubnet(cidr, 16, 0)]
+  }
+  worker_subnets = {
+    ipv4 = [for i, cidr in local.network_cidr.ipv4 : cidrsubnet(cidr, 1, 1)]
+    ipv6 = [for i, cidr in local.network_cidr.ipv6 : cidrsubnet(cidr, 16, 1)]
+  }
+  cluster_subnets = {
+    ipv4 = concat(local.controller_subnets.ipv4, local.worker_subnets.ipv4)
+    ipv6 = concat(local.controller_subnets.ipv6, local.worker_subnets.ipv6)
+  }
+}
+
 # Organize cluster into a resource group
 resource "azurerm_resource_group" "cluster" {
   name     = var.cluster_name
-  location = var.region
+  location = var.location
 }
 
 resource "azurerm_virtual_network" "network" {
+  name                = var.cluster_name
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name          = var.cluster_name
-  location      = azurerm_resource_group.cluster.location
-  address_space = [var.host_cidr]
+  location            = azurerm_resource_group.cluster.location
+  address_space = concat(
+    local.network_cidr.ipv4,
+    local.network_cidr.ipv6
+  )
 }
 
-# Subnets - separate subnets for controller and workers because Azure
-# network security groups are based on IPv4 CIDR rather than instance
-# tags like GCP or security group membership like AWS
+# Subnets - separate subnets for controllers and workers because Azure
+# network security groups are oriented around address prefixes rather
+# than instance tags (GCP) or security group membership (AWS)
 
 resource "azurerm_subnet" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                 = "controller"
+  resource_group_name  = azurerm_resource_group.cluster.name
   virtual_network_name = azurerm_virtual_network.network.name
-  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 0)]
+  address_prefixes = concat(
+    local.controller_subnets.ipv4,
+    local.controller_subnets.ipv6,
+  )
+  default_outbound_access_enabled = false
+
 }
 
 resource "azurerm_subnet_network_security_group_association" "controller" {
@@ -30,11 +67,14 @@ resource "azurerm_subnet_network_security_group_association" "controller" {
 }
 
 resource "azurerm_subnet" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                 = "worker"
+  resource_group_name  = azurerm_resource_group.cluster.name
   virtual_network_name = azurerm_virtual_network.network.name
-  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 1)]
+  address_prefixes = concat(
+    local.worker_subnets.ipv4,
+    local.worker_subnets.ipv6,
+  )
+  default_outbound_access_enabled = false
 }
 
 resource "azurerm_subnet_network_security_group_association" "worker" {

azure/fedora-coreos/kubernetes/outputs.tf

@@ -6,13 +6,18 @@ output "kubeconfig-admin" {
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
-  value       = azurerm_public_ip.ingress-ipv4.ip_address
+  value       = azurerm_public_ip.frontend-ipv4.ip_address
   description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_static_ipv6" {
+  value       = azurerm_public_ip.frontend-ipv6.ip_address
+  description = "IPv6 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
 # Outputs for worker pools
 
-output "region" {
+output "location" {
   value = azurerm_resource_group.cluster.location
 }
 
@@ -51,12 +56,12 @@ output "worker_security_group_name" {
 
 output "controller_address_prefixes" {
   description = "Controller network subnet CIDR addresses (for source/destination)"
-  value       = azurerm_subnet.controller.address_prefixes
+  value       = local.controller_subnets
 }
 
 output "worker_address_prefixes" {
   description = "Worker network subnet CIDR addresses (for source/destination)"
-  value       = azurerm_subnet.worker.address_prefixes
+  value       = local.worker_subnets
 }
 
 # Outputs for custom load balancing
@@ -66,9 +71,12 @@ output "loadbalancer_id" {
   value       = azurerm_lb.cluster.id
 }
 
-output "backend_address_pool_id" {
-  description = "ID of the worker backend address pool"
-  value       = azurerm_lb_backend_address_pool.worker.id
+output "backend_address_pool_ids" {
+  description = "IDs of the worker backend address pools"
+  value = {
+    ipv4 = [azurerm_lb_backend_address_pool.worker-ipv4.id]
+    ipv6 = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  }
 }
 
 # Outputs for debug

azure/fedora-coreos/kubernetes/security.tf

@@ -1,214 +1,223 @@
 # Controller security group
 
 resource "azurerm_network_security_group" "controller" {
+  name                = "${var.cluster_name}-controller"
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = "${var.cluster_name}-controller"
-  location = azurerm_resource_group.cluster.location
+  location            = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "controller-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-icmp"
+  name                         = "allow-icmp-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "1995"
+  priority                     = 1995 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Icmp"
   source_port_range            = "*"
   destination_port_range       = "*"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-ssh"
+  name                         = "allow-ssh-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2000"
+  priority                     = 2000 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "22"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-etcd" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-etcd"
+  name                         = "allow-etcd-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2005"
+  priority                     = 2005 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "2379-2380"
-  source_address_prefixes      = azurerm_subnet.controller.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.controller_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape etcd metrics
 resource "azurerm_network_security_rule" "controller-etcd-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-etcd-metrics"
+  name                         = "allow-etcd-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2010"
+  priority                     = 2010 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "2381"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-proxy metrics
 resource "azurerm_network_security_rule" "controller-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-kube-proxy-metrics"
+  name                         = "allow-kube-proxy-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2011"
+  priority                     = 2012 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10249"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
 resource "azurerm_network_security_rule" "controller-kube-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-kube-metrics"
+  name                         = "allow-kube-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2012"
+  priority                     = 2014 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10257-10259"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-apiserver"
+  name                         = "allow-apiserver-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2015"
+  priority                     = 2016 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "6443"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.controller_subnets : {}
 
-  name                         = "allow-cilium-health"
+  name                         = "allow-cilium-health-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2018"
+  priority                     = 2018 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "4240"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-cilium-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.controller_subnets : {}
 
-  name                         = "allow-cilium-metrics"
+  name                         = "allow-cilium-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2019"
+  priority                     = 2035 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9962-9965"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-vxlan"
+  name                         = "allow-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2020"
+  priority                     = 2020 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "4789"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-linux-vxlan"
+  name                         = "allow-linux-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2021"
+  priority                     = 2022 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "8472"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "controller-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-node-exporter"
+  name                         = "allow-node-exporter-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2025"
+  priority                     = 2025 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9100"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "controller-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-kubelet"
+  name                        = "allow-kubelet-${each.key}"
+  resource_group_name         = azurerm_resource_group.cluster.name
   network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2030"
+  priority                    = 2030 + (each.key == "ipv4" ? 0 : 1)
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "10250"
-
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
@@ -247,182 +256,189 @@ resource "azurerm_network_security_rule" "controller-deny-all" {
 # Worker security group
 
 resource "azurerm_network_security_group" "worker" {
+  name                = "${var.cluster_name}-worker"
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = "${var.cluster_name}-worker"
-  location = azurerm_resource_group.cluster.location
+  location            = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "worker-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-icmp"
+  name                         = "allow-icmp-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "1995"
+  priority                     = 1995 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Icmp"
   source_port_range            = "*"
   destination_port_range       = "*"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-ssh"
+  name                         = "allow-ssh-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2000"
+  priority                     = 2000 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "22"
-  source_address_prefixes      = azurerm_subnet.controller.address_prefixes
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.controller_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-http" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-http"
+  name                         = "allow-http-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2005"
+  priority                     = 2005 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "80"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-https" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-https"
+  name                         = "allow-https-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2010"
+  priority                     = 2010 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "443"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.worker_subnets : {}
 
-  name                         = "allow-cilium-health"
+  name                         = "allow-cilium-health-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2013"
+  priority                     = 2012 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "4240"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-cilium-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.worker_subnets : {}
 
-  name                         = "allow-cilium-metrics"
+  name                         = "allow-cilium-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2014"
+  priority                     = 2014 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9962-9965"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-vxlan"
+  name                         = "allow-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2015"
+  priority                     = 2016 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "4789"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-linux-vxlan"
+  name                         = "allow-linux-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2016"
+  priority                     = 2018 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "8472"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "worker-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-node-exporter"
+  name                         = "allow-node-exporter-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2020"
+  priority                     = 2020 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9100"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-proxy
 resource "azurerm_network_security_rule" "worker-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-kube-proxy"
+  name                         = "allow-kube-proxy-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2024"
+  priority                     = 2024 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10249"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "worker-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-kubelet"
+  name                        = "allow-kubelet-${each.key}"
+  resource_group_name         = azurerm_resource_group.cluster.name
   network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2025"
+  priority                    = 2026 + (each.key == "ipv4" ? 0 : 1)
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "10250"
-
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound

azure/fedora-coreos/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
 
   connection {
     type    = "ssh"
-    host    = azurerm_public_ip.controllers.*.ip_address[count.index]
+    host    = azurerm_public_ip.controllers-ipv4[count.index].ip_address
     user    = "core"
     timeout = "15m"
   }
@@ -45,7 +45,7 @@ resource "null_resource" "bootstrap" {
 
   connection {
     type    = "ssh"
-    host    = azurerm_public_ip.controllers.*.ip_address[0]
+    host    = azurerm_public_ip.controllers-ipv4[0].ip_address
     user    = "core"
     timeout = "15m"
   }

azure/fedora-coreos/kubernetes/variables.tf

@@ -5,9 +5,9 @@ variable "cluster_name" {
 
 # Azure
 
-variable "region" {
+variable "location" {
   type        = string
-  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+  description = "Azure location (e.g. centralus , see `az account list-locations --output table`)"
 }
 
 variable "dns_zone" {
@@ -22,41 +22,66 @@ variable "dns_zone_group" {
 
 # instances
 
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for instances"
+}
+
+
 variable "controller_count" {
   type        = number
   description = "Number of controllers (i.e. masters)"
   default     = 1
 }
 
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
 variable "controller_type" {
   type        = string
   description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
   default     = "Standard_B2s"
 }
 
+variable "controller_disk_type" {
+  type        = string
+  description = "Type of managed disk for controller node(s)"
+  default     = "Premium_LRS"
+}
+
+variable "controller_disk_size" {
+  type        = number
+  description = "Size of the managed disk in GB for controller node(s)"
+  default     = 30
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
 variable "worker_type" {
   type        = string
   description = "Machine type for workers (see `az vm list-skus --location centralus`)"
   default     = "Standard_D2as_v5"
 }
 
-variable "os_image" {
+variable "worker_disk_type" {
   type        = string
-  description = "Fedora CoreOS image for instances"
+  description = "Type of managed disk for worker nodes"
+  default     = "Standard_LRS"
 }
 
-variable "disk_size" {
+variable "worker_disk_size" {
   type        = number
-  description = "Size of the disk in GB"
+  description = "Size of the managed disk in GB for worker nodes"
   default     = 30
 }
 
+variable "worker_ephemeral_disk" {
+  type        = bool
+  description = "Use ephemeral local disk instead of managed disk (requires vm_type with local storage)"
+  default     = false
+}
+
 variable "worker_priority" {
   type        = string
   description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
@@ -94,10 +119,15 @@ variable "networking" {
   default     = "cilium"
 }
 
-variable "host_cidr" {
-  type        = string
-  description = "CIDR IPv4 range to assign to instances"
-  default     = "10.0.0.0/16"
+variable "network_cidr" {
+  type = object({
+    ipv4 = list(string)
+    ipv6 = optional(list(string), [])
+  })
+  description = "Virtual network CIDR ranges"
+  default = {
+    ipv4 = ["10.0.0.0/16"]
+  }
 }
 
 variable "pod_cidr" {

azure/fedora-coreos/kubernetes/workers.tf

@@ -3,16 +3,19 @@ module "workers" {
   name   = var.cluster_name
 
   # Azure
-  resource_group_name     = azurerm_resource_group.cluster.name
-  region                  = azurerm_resource_group.cluster.location
-  subnet_id               = azurerm_subnet.worker.id
-  security_group_id       = azurerm_network_security_group.worker.id
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  resource_group_name      = azurerm_resource_group.cluster.name
+  location                 = azurerm_resource_group.cluster.location
+  subnet_id                = azurerm_subnet.worker.id
+  security_group_id        = azurerm_network_security_group.worker.id
+  backend_address_pool_ids = local.backend_address_pool_ids
 
-  worker_count = var.worker_count
-  vm_type      = var.worker_type
-  os_image     = var.os_image
-  priority     = var.worker_priority
+  worker_count   = var.worker_count
+  vm_type        = var.worker_type
+  os_image       = var.os_image
+  disk_type      = var.worker_disk_type
+  disk_size      = var.worker_disk_size
+  ephemeral_disk = var.worker_ephemeral_disk
+  priority       = var.worker_priority
 
   # configuration
   kubeconfig            = module.bootstrap.kubeconfig-kubelet

azure/fedora-coreos/kubernetes/workers/butane/worker.yaml

@@ -26,7 +26,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin

azure/fedora-coreos/kubernetes/workers/variables.tf

@@ -5,9 +5,9 @@ variable "name" {
 
 # Azure
 
-variable "region" {
+variable "location" {
   type        = string
-  description = "Must be set to the Azure Region of cluster"
+  description = "Must be set to the Azure location of cluster"
 }
 
 variable "resource_group_name" {
@@ -25,9 +25,12 @@ variable "security_group_id" {
   description = "Must be set to the `worker_security_group_id` output by cluster"
 }
 
-variable "backend_address_pool_id" {
-  type        = string
-  description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
+variable "backend_address_pool_ids" {
+  type = object({
+    ipv4 = list(string)
+    ipv6 = list(string)
+  })
+  description = "Must be set to the `backend_address_pool_ids` output by cluster"
 }
 
 # instances
@@ -49,6 +52,24 @@ variable "os_image" {
   description = "Fedora CoreOS image for instances"
 }
 
+variable "disk_type" {
+  type        = string
+  description = "Type of managed disk"
+  default     = "Standard_LRS"
+}
+
+variable "disk_size" {
+  type        = number
+  description = "Size of the managed disk in GB"
+  default     = 30
+}
+
+variable "ephemeral_disk" {
+  type        = bool
+  description = "Use ephemeral local disk instead of managed disk (requires vm_type with local storage)"
+  default     = false
+}
+
 variable "priority" {
   type        = string
   description = "Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."

azure/fedora-coreos/kubernetes/workers/workers.tf

@@ -3,30 +3,29 @@ locals {
 }
 
 # Workers scale set
-resource "azurerm_linux_virtual_machine_scale_set" "workers" {
-  resource_group_name = var.resource_group_name
-
-  name      = "${var.name}-worker"
-  location  = var.region
-  sku       = var.vm_type
-  instances = var.worker_count
-  # instance name prefix for instances in the set
-  computer_name_prefix   = "${var.name}-worker"
-  single_placement_group = false
-  custom_data            = base64encode(data.ct_config.worker.rendered)
+resource "azurerm_orchestrated_virtual_machine_scale_set" "workers" {
+  name                        = "${var.name}-worker"
+  resource_group_name         = var.resource_group_name
+  location                    = var.location
+  platform_fault_domain_count = 1
+  sku_name                    = var.vm_type
+  instances                   = var.worker_count
 
   # storage
-  source_image_id = var.os_image
+  encryption_at_host_enabled = true
+  source_image_id            = var.os_image
   os_disk {
-    storage_account_type = "Standard_LRS"
-    caching              = "ReadWrite"
-  }
-
-  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
-  admin_username = "core"
-  admin_ssh_key {
-    username   = "core"
-    public_key = var.azure_authorized_key
+    storage_account_type = var.disk_type
+    disk_size_gb         = var.disk_size
+    caching              = "ReadOnly"
+    # Optionally, use the ephemeral disk of the instance type (support varies)
+    dynamic "diff_disk_settings" {
+      for_each = var.ephemeral_disk ? [1] : []
+      content {
+        option    = "Local"
+        placement = "ResourceDisk"
+      }
+    }
   }
 
   # network
@@ -36,41 +35,46 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
     network_security_group_id = var.security_group_id
 
     ip_configuration {
-      name      = "ip0"
+      name      = "ipv4"
+      version   = "IPv4"
       primary   = true
       subnet_id = var.subnet_id
-
       # backend address pool to which the NIC should be added
-      load_balancer_backend_address_pool_ids = [var.backend_address_pool_id]
+      load_balancer_backend_address_pool_ids = var.backend_address_pool_ids.ipv4
+    }
+    ip_configuration {
+      name      = "ipv6"
+      version   = "IPv6"
+      subnet_id = var.subnet_id
+      # backend address pool to which the NIC should be added
+      load_balancer_backend_address_pool_ids = var.backend_address_pool_ids.ipv6
+    }
+  }
+
+  # boot
+  user_data_base64 = base64encode(data.ct_config.worker.rendered)
+  boot_diagnostics {
+    # defaults to a managed storage account
+  }
+
+  # Azure requires an RSA admin_ssh_key
+  os_profile {
+    linux_configuration {
+      admin_username = "core"
+      admin_ssh_key {
+        username   = "core"
+        public_key = local.azure_authorized_key
+      }
+      computer_name_prefix = "${var.name}-worker"
     }
   }
 
   # lifecycle
-  upgrade_mode = "Manual"
   # eviction policy may only be set when priority is Spot
   priority        = var.priority
   eviction_policy = var.priority == "Spot" ? "Delete" : null
-}
-
-# Scale up or down to maintain desired number, tolerating deallocations.
-resource "azurerm_monitor_autoscale_setting" "workers" {
-  resource_group_name = var.resource_group_name
-
-  name     = "${var.name}-maintain-desired"
-  location = var.region
-
-  # autoscale
-  enabled            = true
-  target_resource_id = azurerm_linux_virtual_machine_scale_set.workers.id
-
-  profile {
-    name = "default"
-
-    capacity {
-      minimum = var.worker_count
-      default = var.worker_count
-      maximum = var.worker_count
-    }
+  termination_notification {
+    enabled = true
   }
 }
 

azure/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

azure/flatcar-linux/kubernetes/butane/controller.yaml

@@ -56,7 +56,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -105,7 +105,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

azure/flatcar-linux/kubernetes/controllers.tf

@@ -1,25 +1,9 @@
-# Discrete DNS records for each controller's private IPv4 for etcd usage
-resource "azurerm_dns_a_record" "etcds" {
-  count               = var.controller_count
-  resource_group_name = var.dns_zone_group
-
-  # DNS Zone name where record should be created
-  zone_name = var.dns_zone
-
-  # DNS record
-  name = format("%s-etcd%d", var.cluster_name, count.index)
-  ttl  = 300
-
-  # private IPv4 address for etcd
-  records = [azurerm_network_interface.controllers.*.private_ip_address[count.index]]
-}
-
 locals {
   # Container Linux derivative
   # flatcar-stable -> Flatcar Linux Stable
   channel      = split("-", var.os_image)[1]
-  offer_suffix = var.arch == "arm64" ? "corevm" : "free"
-  urn          = var.arch == "arm64" ? local.channel : "${local.channel}-gen2"
+  offer_suffix = var.controller_arch == "arm64" ? "corevm" : "free"
+  urn          = var.controller_arch == "arm64" ? local.channel : "${local.channel}-gen2"
 
   # Typhoon ssh_authorized_key supports RSA or a newer formats (e.g. ed25519).
   # However, Azure requires an older RSA key to pass validations. To use a
@@ -28,12 +12,25 @@ locals {
   azure_authorized_key = var.azure_authorized_key == "" ? var.ssh_authorized_key : var.azure_authorized_key
 }
 
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "azurerm_dns_a_record" "etcds" {
+  count = var.controller_count
+
+  # DNS Zone name where record should be created
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
+  # DNS record
+  name = format("%s-etcd%d", var.cluster_name, count.index)
+  ttl  = 300
+  # private IPv4 address for etcd
+  records = [azurerm_network_interface.controllers[count.index].private_ip_address]
+}
+
 # Controller availability set to spread controllers
 resource "azurerm_availability_set" "controllers" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                         = "${var.cluster_name}-controllers"
-  location                     = var.region
+  resource_group_name          = azurerm_resource_group.cluster.name
+  location                     = var.location
   platform_fault_domain_count  = 2
   platform_update_domain_count = 4
   managed                      = true
@@ -41,25 +38,20 @@ resource "azurerm_availability_set" "controllers" {
 
 # Controller instances
 resource "azurerm_linux_virtual_machine" "controllers" {
-  count               = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
+  count = var.controller_count
 
   name                = "${var.cluster_name}-controller-${count.index}"
-  location            = var.region
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
   availability_set_id = azurerm_availability_set.controllers.id
-
-  size        = var.controller_type
-  custom_data = base64encode(data.ct_config.controllers.*.rendered[count.index])
-  boot_diagnostics {
-    # defaults to a managed storage account
-  }
+  size                = var.controller_type
 
   # storage
   os_disk {
     name                 = "${var.cluster_name}-controller-${count.index}"
+    storage_account_type = var.controller_disk_type
+    disk_size_gb         = var.controller_disk_size
     caching              = "None"
-    disk_size_gb         = var.disk_size
-    storage_account_type = "Premium_LRS"
   }
 
   # Flatcar Container Linux
@@ -71,7 +63,7 @@ resource "azurerm_linux_virtual_machine" "controllers" {
   }
 
   dynamic "plan" {
-    for_each = var.arch == "arm64" ? [] : [1]
+    for_each = var.controller_arch == "arm64" ? [] : [1]
     content {
       publisher = "kinvolk"
       product   = "flatcar-container-linux-${local.offer_suffix}"
@@ -84,7 +76,13 @@ resource "azurerm_linux_virtual_machine" "controllers" {
     azurerm_network_interface.controllers[count.index].id
   ]
 
-  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
+  # boot
+  custom_data = base64encode(data.ct_config.controllers[count.index].rendered)
+  boot_diagnostics {
+    # defaults to a managed storage account
+  }
+
+  # Azure requires an RSA admin_ssh_key
   admin_username = "core"
   admin_ssh_key {
     username   = "core"
@@ -99,31 +97,52 @@ resource "azurerm_linux_virtual_machine" "controllers" {
   }
 }
 
-# Controller public IPv4 addresses
-resource "azurerm_public_ip" "controllers" {
-  count               = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
+# Controller node public IPv4 addresses
+resource "azurerm_public_ip" "controllers-ipv4" {
+  count = var.controller_count
 
-  name              = "${var.cluster_name}-controller-${count.index}"
-  location          = azurerm_resource_group.cluster.location
-  sku               = "Standard"
-  allocation_method = "Static"
+  name                = "${var.cluster_name}-controller-${count.index}-ipv4"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
+  ip_version          = "IPv4"
+  sku                 = "Standard"
+  allocation_method   = "Static"
 }
 
-# Controller NICs with public and private IPv4
-resource "azurerm_network_interface" "controllers" {
-  count               = var.controller_count
-  resource_group_name = azurerm_resource_group.cluster.name
+# Controller node public IPv6 addresses
+resource "azurerm_public_ip" "controllers-ipv6" {
+  count = var.controller_count
 
-  name     = "${var.cluster_name}-controller-${count.index}"
-  location = azurerm_resource_group.cluster.location
+  name                = "${var.cluster_name}-controller-${count.index}-ipv6"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
+  ip_version          = "IPv6"
+  sku                 = "Standard"
+  allocation_method   = "Static"
+}
+
+# Controllers' network interfaces
+resource "azurerm_network_interface" "controllers" {
+  count = var.controller_count
+
+  name                = "${var.cluster_name}-controller-${count.index}"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = azurerm_resource_group.cluster.location
 
   ip_configuration {
-    name                          = "ip0"
+    name                          = "ipv4"
+    primary                       = true
     subnet_id                     = azurerm_subnet.controller.id
     private_ip_address_allocation = "Dynamic"
-    # instance public IPv4
-    public_ip_address_id = azurerm_public_ip.controllers.*.id[count.index]
+    private_ip_address_version    = "IPv4"
+    public_ip_address_id          = azurerm_public_ip.controllers-ipv4[count.index].id
+  }
+  ip_configuration {
+    name                          = "ipv6"
+    subnet_id                     = azurerm_subnet.controller.id
+    private_ip_address_allocation = "Dynamic"
+    private_ip_address_version    = "IPv6"
+    public_ip_address_id          = azurerm_public_ip.controllers-ipv6[count.index].id
   }
 }
 
@@ -135,13 +154,21 @@ resource "azurerm_network_interface_security_group_association" "controllers" {
   network_security_group_id = azurerm_network_security_group.controller.id
 }
 
-# Associate controller network interface with controller backend address pool
-resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+# Associate controller network interface with controller backend address pools
+resource "azurerm_network_interface_backend_address_pool_association" "controllers-ipv4" {
   count = var.controller_count
 
   network_interface_id    = azurerm_network_interface.controllers[count.index].id
-  ip_configuration_name   = "ip0"
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  ip_configuration_name   = "ipv4"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller-ipv4.id
+}
+
+resource "azurerm_network_interface_backend_address_pool_association" "controllers-ipv6" {
+  count = var.controller_count
+
+  network_interface_id    = azurerm_network_interface.controllers[count.index].id
+  ip_configuration_name   = "ipv6"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller-ipv6.id
 }
 
 # Flatcar Linux controllers

azure/flatcar-linux/kubernetes/lb.tf

@@ -1,116 +1,164 @@
-# DNS record for the apiserver load balancer
+# DNS A record for the apiserver load balancer
 resource "azurerm_dns_a_record" "apiserver" {
-  resource_group_name = var.dns_zone_group
-
   # DNS Zone name where record should be created
-  zone_name = var.dns_zone
-
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
   # DNS record
   name = var.cluster_name
   ttl  = 300
-
   # IPv4 address of apiserver load balancer
-  records = [azurerm_public_ip.apiserver-ipv4.ip_address]
+  records = [azurerm_public_ip.frontend-ipv4.ip_address]
 }
 
-# Static IPv4 address for the apiserver frontend
-resource "azurerm_public_ip" "apiserver-ipv4" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name              = "${var.cluster_name}-apiserver-ipv4"
-  location          = var.region
-  sku               = "Standard"
-  allocation_method = "Static"
+# DNS AAAA record for the apiserver load balancer
+resource "azurerm_dns_aaaa_record" "apiserver" {
+  # DNS Zone name where record should be created
+  zone_name           = var.dns_zone
+  resource_group_name = var.dns_zone_group
+  # DNS record
+  name = var.cluster_name
+  ttl  = 300
+  # IPv6 address of apiserver load balancer
+  records = [azurerm_public_ip.frontend-ipv6.ip_address]
 }
 
-# Static IPv4 address for the ingress frontend
-resource "azurerm_public_ip" "ingress-ipv4" {
+# Static IPv4 address for the load balancer
+resource "azurerm_public_ip" "frontend-ipv4" {
+  name                = "${var.cluster_name}-frontend-ipv4"
   resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
+  ip_version          = "IPv4"
+  sku                 = "Standard"
+  allocation_method   = "Static"
+}
 
-  name              = "${var.cluster_name}-ingress-ipv4"
-  location          = var.region
-  sku               = "Standard"
-  allocation_method = "Static"
+# Static IPv6 address for the load balancer
+resource "azurerm_public_ip" "frontend-ipv6" {
+  name                = "${var.cluster_name}-ingress-ipv6"
+  resource_group_name = azurerm_resource_group.cluster.name
+  location            = var.location
+  ip_version          = "IPv6"
+  sku                 = "Standard"
+  allocation_method   = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress
 resource "azurerm_lb" "cluster" {
+  name                = var.cluster_name
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = var.cluster_name
-  location = var.region
-  sku      = "Standard"
+  location            = var.location
+  sku                 = "Standard"
 
   frontend_ip_configuration {
-    name                 = "apiserver"
-    public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
+    name                 = "frontend-ipv4"
+    public_ip_address_id = azurerm_public_ip.frontend-ipv4.id
   }
 
   frontend_ip_configuration {
-    name                 = "ingress"
-    public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
+    name                 = "frontend-ipv6"
+    public_ip_address_id = azurerm_public_ip.frontend-ipv6.id
   }
 }
 
-resource "azurerm_lb_rule" "apiserver" {
-  name                           = "apiserver"
+resource "azurerm_lb_rule" "apiserver-ipv4" {
+  name                           = "apiserver-ipv4"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "apiserver"
+  frontend_ip_configuration_name = "frontend-ipv4"
+  disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 6443
   backend_port             = 6443
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller-ipv4.id]
   probe_id                 = azurerm_lb_probe.apiserver.id
 }
 
-resource "azurerm_lb_rule" "ingress-http" {
-  name                           = "ingress-http"
+resource "azurerm_lb_rule" "apiserver-ipv6" {
+  name                           = "apiserver-ipv6"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller-ipv6.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
+}
+
+resource "azurerm_lb_rule" "ingress-http-ipv4" {
+  name                           = "ingress-http-ipv4"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv4"
   disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 80
   backend_port             = 80
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv4.id]
   probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-resource "azurerm_lb_rule" "ingress-https" {
-  name                           = "ingress-https"
+resource "azurerm_lb_rule" "ingress-https-ipv4" {
+  name                           = "ingress-https-ipv4"
   loadbalancer_id                = azurerm_lb.cluster.id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "frontend-ipv4"
   disable_outbound_snat          = true
 
   protocol                 = "Tcp"
   frontend_port            = 443
   backend_port             = 443
-  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv4.id]
   probe_id                 = azurerm_lb_probe.ingress.id
 }
 
-# Worker outbound TCP/UDP SNAT
-resource "azurerm_lb_outbound_rule" "worker-outbound" {
-  name            = "worker"
-  loadbalancer_id = azurerm_lb.cluster.id
-  frontend_ip_configuration {
-    name = "ingress"
-  }
+resource "azurerm_lb_rule" "ingress-http-ipv6" {
+  name                           = "ingress-http-ipv6"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
 
-  protocol                = "All"
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }
 
+resource "azurerm_lb_rule" "ingress-https-ipv6" {
+  name                           = "ingress-https-ipv6"
+  loadbalancer_id                = azurerm_lb.cluster.id
+  frontend_ip_configuration_name = "frontend-ipv6"
+  disable_outbound_snat          = true
+
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
+}
+
+# Backend Address Pools
+
 # Address pool of controllers
-resource "azurerm_lb_backend_address_pool" "controller" {
-  name            = "controller"
+resource "azurerm_lb_backend_address_pool" "controller-ipv4" {
+  name            = "controller-ipv4"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
-# Address pool of workers
-resource "azurerm_lb_backend_address_pool" "worker" {
-  name            = "worker"
+resource "azurerm_lb_backend_address_pool" "controller-ipv6" {
+  name            = "controller-ipv6"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+# Address pools for workers
+resource "azurerm_lb_backend_address_pool" "worker-ipv4" {
+  name            = "worker-ipv4"
+  loadbalancer_id = azurerm_lb.cluster.id
+}
+
+resource "azurerm_lb_backend_address_pool" "worker-ipv6" {
+  name            = "worker-ipv6"
   loadbalancer_id = azurerm_lb.cluster.id
 }
 
@@ -122,10 +170,8 @@ resource "azurerm_lb_probe" "apiserver" {
   loadbalancer_id = azurerm_lb.cluster.id
   protocol        = "Tcp"
   port            = 6443
-
   # unhealthy threshold
-  number_of_probes = 3
-
+  number_of_probes    = 3
   interval_in_seconds = 5
 }
 
@@ -136,10 +182,29 @@ resource "azurerm_lb_probe" "ingress" {
   protocol        = "Http"
   port            = 10254
   request_path    = "/healthz"
-
   # unhealthy threshold
-  number_of_probes = 3
-
+  number_of_probes    = 3
   interval_in_seconds = 5
 }
 
+# Outbound SNAT
+
+resource "azurerm_lb_outbound_rule" "outbound-ipv4" {
+  name                    = "outbound-ipv4"
+  protocol                = "All"
+  loadbalancer_id         = azurerm_lb.cluster.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker-ipv4.id
+  frontend_ip_configuration {
+    name = "frontend-ipv4"
+  }
+}
+
+resource "azurerm_lb_outbound_rule" "outbound-ipv6" {
+  name                    = "outbound-ipv6"
+  protocol                = "All"
+  loadbalancer_id         = azurerm_lb.cluster.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker-ipv6.id
+  frontend_ip_configuration {
+    name = "frontend-ipv6"
+  }
+}

azure/flatcar-linux/kubernetes/locals.tf

@@ -0,0 +1,6 @@
+locals {
+  backend_address_pool_ids = {
+    ipv4 = [azurerm_lb_backend_address_pool.worker-ipv4.id]
+    ipv6 = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  }
+}

azure/flatcar-linux/kubernetes/network.tf

@@ -1,27 +1,63 @@
+# Choose an IPv6 ULA subnet at random
+# https://datatracker.ietf.org/doc/html/rfc4193
+resource "random_id" "ula-netnum" {
+  byte_length = 5 # 40 bits
+}
+
+locals {
+  # fd00::/8 -> shift 40 -> 2^40 possible /48 subnets
+  ula-range = cidrsubnet("fd00::/8", 40, random_id.ula-netnum.dec)
+  network_cidr = {
+    ipv4 = var.network_cidr.ipv4
+    ipv6 = length(var.network_cidr.ipv6) > 0 ? var.network_cidr.ipv6 : [local.ula-range]
+  }
+
+  # Subdivide the virtual network into subnets
+  # - controllers use netnum 0
+  # - workers use netnum 1
+  controller_subnets = {
+    ipv4 = [for i, cidr in local.network_cidr.ipv4 : cidrsubnet(cidr, 1, 0)]
+    ipv6 = [for i, cidr in local.network_cidr.ipv6 : cidrsubnet(cidr, 16, 0)]
+  }
+  worker_subnets = {
+    ipv4 = [for i, cidr in local.network_cidr.ipv4 : cidrsubnet(cidr, 1, 1)]
+    ipv6 = [for i, cidr in local.network_cidr.ipv6 : cidrsubnet(cidr, 16, 1)]
+  }
+  cluster_subnets = {
+    ipv4 = concat(local.controller_subnets.ipv4, local.worker_subnets.ipv4)
+    ipv6 = concat(local.controller_subnets.ipv6, local.worker_subnets.ipv6)
+  }
+}
+
 # Organize cluster into a resource group
 resource "azurerm_resource_group" "cluster" {
   name     = var.cluster_name
-  location = var.region
+  location = var.location
 }
 
 resource "azurerm_virtual_network" "network" {
+  name                = var.cluster_name
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name          = var.cluster_name
-  location      = azurerm_resource_group.cluster.location
-  address_space = [var.host_cidr]
+  location            = azurerm_resource_group.cluster.location
+  address_space = concat(
+    local.network_cidr.ipv4,
+    local.network_cidr.ipv6
+  )
 }
 
-# Subnets - separate subnets for controller and workers because Azure
-# network security groups are based on IPv4 CIDR rather than instance
-# tags like GCP or security group membership like AWS
+# Subnets - separate subnets for controllers and workers because Azure
+# network security groups are oriented around address prefixes rather
+# than instance tags (GCP) or security group membership (AWS)
 
 resource "azurerm_subnet" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                 = "controller"
+  resource_group_name  = azurerm_resource_group.cluster.name
   virtual_network_name = azurerm_virtual_network.network.name
-  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 0)]
+  address_prefixes = concat(
+    local.controller_subnets.ipv4,
+    local.controller_subnets.ipv6,
+  )
+  default_outbound_access_enabled = false
 }
 
 resource "azurerm_subnet_network_security_group_association" "controller" {
@@ -30,11 +66,14 @@ resource "azurerm_subnet_network_security_group_association" "controller" {
 }
 
 resource "azurerm_subnet" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
   name                 = "worker"
+  resource_group_name  = azurerm_resource_group.cluster.name
   virtual_network_name = azurerm_virtual_network.network.name
-  address_prefixes     = [cidrsubnet(var.host_cidr, 1, 1)]
+  address_prefixes = concat(
+    local.worker_subnets.ipv4,
+    local.worker_subnets.ipv6,
+  )
+  default_outbound_access_enabled = false
 }
 
 resource "azurerm_subnet_network_security_group_association" "worker" {

azure/flatcar-linux/kubernetes/outputs.tf

@@ -6,13 +6,18 @@ output "kubeconfig-admin" {
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
-  value       = azurerm_public_ip.ingress-ipv4.ip_address
+  value       = azurerm_public_ip.frontend-ipv4.ip_address
   description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_static_ipv6" {
+  value       = azurerm_public_ip.frontend-ipv6.ip_address
+  description = "IPv6 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
 # Outputs for worker pools
 
-output "region" {
+output "location" {
   value = azurerm_resource_group.cluster.location
 }
 
@@ -51,12 +56,12 @@ output "worker_security_group_name" {
 
 output "controller_address_prefixes" {
   description = "Controller network subnet CIDR addresses (for source/destination)"
-  value       = azurerm_subnet.controller.address_prefixes
+  value       = local.controller_subnets
 }
 
 output "worker_address_prefixes" {
   description = "Worker network subnet CIDR addresses (for source/destination)"
-  value       = azurerm_subnet.worker.address_prefixes
+  value       = local.worker_subnets
 }
 
 # Outputs for custom load balancing
@@ -66,9 +71,12 @@ output "loadbalancer_id" {
   value       = azurerm_lb.cluster.id
 }
 
-output "backend_address_pool_id" {
-  description = "ID of the worker backend address pool"
-  value       = azurerm_lb_backend_address_pool.worker.id
+output "backend_address_pool_ids" {
+  description = "IDs of the worker backend address pools"
+  value = {
+    ipv4 = [azurerm_lb_backend_address_pool.worker-ipv4.id]
+    ipv6 = [azurerm_lb_backend_address_pool.worker-ipv6.id]
+  }
 }
 
 # Outputs for debug

azure/flatcar-linux/kubernetes/security.tf

@@ -1,214 +1,223 @@
 # Controller security group
 
 resource "azurerm_network_security_group" "controller" {
+  name                = "${var.cluster_name}-controller"
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = "${var.cluster_name}-controller"
-  location = azurerm_resource_group.cluster.location
+  location            = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "controller-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-icmp"
+  name                         = "allow-icmp-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "1995"
+  priority                     = 1995 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Icmp"
   source_port_range            = "*"
   destination_port_range       = "*"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-ssh"
+  name                         = "allow-ssh-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2000"
+  priority                     = 2000 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "22"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-etcd" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-etcd"
+  name                         = "allow-etcd-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2005"
+  priority                     = 2005 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "2379-2380"
-  source_address_prefixes      = azurerm_subnet.controller.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.controller_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape etcd metrics
 resource "azurerm_network_security_rule" "controller-etcd-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-etcd-metrics"
+  name                         = "allow-etcd-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2010"
+  priority                     = 2010 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "2381"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-proxy metrics
 resource "azurerm_network_security_rule" "controller-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-kube-proxy-metrics"
+  name                         = "allow-kube-proxy-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2011"
+  priority                     = 2012 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10249"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
 resource "azurerm_network_security_rule" "controller-kube-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-kube-metrics"
+  name                         = "allow-kube-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2012"
+  priority                     = 2014 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10257-10259"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-apiserver"
+  name                         = "allow-apiserver-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2015"
+  priority                     = 2016 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "6443"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.controller_subnets : {}
 
-  name                         = "allow-cilium-health"
+  name                         = "allow-cilium-health-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2018"
+  priority                     = 2018 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "4240"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-cilium-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.controller_subnets : {}
 
-  name                         = "allow-cilium-metrics"
+  name                         = "allow-cilium-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2019"
+  priority                     = 2035 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9962-9965"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-vxlan"
+  name                         = "allow-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2020"
+  priority                     = 2020 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "4789"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "controller-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-linux-vxlan"
+  name                         = "allow-linux-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2021"
+  priority                     = 2022 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "8472"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "controller-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                         = "allow-node-exporter"
+  name                         = "allow-node-exporter-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.controller.name
-  priority                     = "2025"
+  priority                     = 2025 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9100"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "controller-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.controller_subnets
 
-  name                        = "allow-kubelet"
+  name                        = "allow-kubelet-${each.key}"
+  resource_group_name         = azurerm_resource_group.cluster.name
   network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2030"
+  priority                    = 2030 + (each.key == "ipv4" ? 0 : 1)
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "10250"
-
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.controller_subnets[each.key]
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
@@ -247,182 +256,189 @@ resource "azurerm_network_security_rule" "controller-deny-all" {
 # Worker security group
 
 resource "azurerm_network_security_group" "worker" {
+  name                = "${var.cluster_name}-worker"
   resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = "${var.cluster_name}-worker"
-  location = azurerm_resource_group.cluster.location
+  location            = azurerm_resource_group.cluster.location
 }
 
 resource "azurerm_network_security_rule" "worker-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-icmp"
+  name                         = "allow-icmp-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "1995"
+  priority                     = 1995 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Icmp"
   source_port_range            = "*"
   destination_port_range       = "*"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-ssh"
+  name                         = "allow-ssh-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2000"
+  priority                     = 2000 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "22"
-  source_address_prefixes      = azurerm_subnet.controller.address_prefixes
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.controller_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-http" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-http"
+  name                         = "allow-http-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2005"
+  priority                     = 2005 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "80"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-https" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-https"
+  name                         = "allow-https-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2010"
+  priority                     = 2010 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "443"
   source_address_prefix        = "*"
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.worker_subnets : {}
 
-  name                         = "allow-cilium-health"
+  name                         = "allow-cilium-health-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2013"
+  priority                     = 2012 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "4240"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-cilium-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
+  for_each = var.networking == "cilium" ? local.worker_subnets : {}
 
-  name                         = "allow-cilium-metrics"
+  name                         = "allow-cilium-metrics-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2014"
+  priority                     = 2014 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9962-9965"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-vxlan"
+  name                         = "allow-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2015"
+  priority                     = 2016 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "4789"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 resource "azurerm_network_security_rule" "worker-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-linux-vxlan"
+  name                         = "allow-linux-vxlan-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2016"
+  priority                     = 2018 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Udp"
   source_port_range            = "*"
   destination_port_range       = "8472"
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "worker-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-node-exporter"
+  name                         = "allow-node-exporter-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2020"
+  priority                     = 2020 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "9100"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow Prometheus to scrape kube-proxy
 resource "azurerm_network_security_rule" "worker-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                         = "allow-kube-proxy"
+  name                         = "allow-kube-proxy-${each.key}"
+  resource_group_name          = azurerm_resource_group.cluster.name
   network_security_group_name  = azurerm_network_security_group.worker.name
-  priority                     = "2024"
+  priority                     = 2024 + (each.key == "ipv4" ? 0 : 1)
   access                       = "Allow"
   direction                    = "Inbound"
   protocol                     = "Tcp"
   source_port_range            = "*"
   destination_port_range       = "10249"
-  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.worker_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "worker-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
+  for_each = local.worker_subnets
 
-  name                        = "allow-kubelet"
+  name                        = "allow-kubelet-${each.key}"
+  resource_group_name         = azurerm_resource_group.cluster.name
   network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2025"
+  priority                    = 2026 + (each.key == "ipv4" ? 0 : 1)
   access                      = "Allow"
   direction                   = "Inbound"
   protocol                    = "Tcp"
   source_port_range           = "*"
   destination_port_range      = "10250"
-
   # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
-  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
+  source_address_prefixes      = local.cluster_subnets[each.key]
+  destination_address_prefixes = local.worker_subnets[each.key]
 }
 
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound

azure/flatcar-linux/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
 
   connection {
     type    = "ssh"
-    host    = azurerm_public_ip.controllers.*.ip_address[count.index]
+    host    = azurerm_public_ip.controllers-ipv4[count.index].ip_address
     user    = "core"
     timeout = "15m"
   }
@@ -45,7 +45,7 @@ resource "null_resource" "bootstrap" {
 
   connection {
     type    = "ssh"
-    host    = azurerm_public_ip.controllers.*.ip_address[0]
+    host    = azurerm_public_ip.controllers-ipv4[0].ip_address
     user    = "core"
     timeout = "15m"
   }

azure/flatcar-linux/kubernetes/variables.tf

@@ -5,9 +5,9 @@ variable "cluster_name" {
 
 # Azure
 
-variable "region" {
+variable "location" {
   type        = string
-  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+  description = "Azure location (e.g. centralus , see `az account list-locations --output table`)"
 }
 
 variable "dns_zone" {
@@ -22,30 +22,6 @@ variable "dns_zone_group" {
 
 # instances
 
-variable "controller_count" {
-  type        = number
-  description = "Number of controllers (i.e. masters)"
-  default     = 1
-}
-
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
-variable "controller_type" {
-  type        = string
-  description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
-  default     = "Standard_B2s"
-}
-
-variable "worker_type" {
-  type        = string
-  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
-  default     = "Standard_D2as_v5"
-}
-
 variable "os_image" {
   type        = string
   description = "Channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
@@ -57,12 +33,60 @@ variable "os_image" {
   }
 }
 
-variable "disk_size" {
+variable "controller_count" {
   type        = number
-  description = "Size of the disk in GB"
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
+  default     = "Standard_B2s"
+}
+
+variable "controller_disk_type" {
+  type        = string
+  description = "Type of managed disk for controller node(s)"
+  default     = "Premium_LRS"
+}
+
+variable "controller_disk_size" {
+  type        = number
+  description = "Size of the managed disk in GB for controller node(s)"
   default     = 30
 }
 
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "worker_type" {
+  type        = string
+  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
+  default     = "Standard_D2as_v5"
+}
+
+variable "worker_disk_type" {
+  type        = string
+  description = "Type of managed disk for worker nodes"
+  default     = "Standard_LRS"
+}
+
+variable "worker_disk_size" {
+  type        = number
+  description = "Size of the managed disk in GB for worker nodes"
+  default     = 30
+}
+
+variable "worker_ephemeral_disk" {
+  type        = bool
+  description = "Use ephemeral local disk instead of managed disk (requires vm_type with local storage)"
+  default     = false
+}
+
 variable "worker_priority" {
   type        = string
   description = "Set worker priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
@@ -100,10 +124,15 @@ variable "networking" {
   default     = "cilium"
 }
 
-variable "host_cidr" {
-  type        = string
-  description = "CIDR IPv4 range to assign to instances"
-  default     = "10.0.0.0/16"
+variable "network_cidr" {
+  type = object({
+    ipv4 = list(string)
+    ipv6 = optional(list(string), [])
+  })
+  description = "Virtual network CIDR ranges"
+  default = {
+    ipv4 = ["10.0.0.0/16"]
+  }
 }
 
 variable "pod_cidr" {
@@ -139,14 +168,25 @@ variable "worker_node_labels" {
   default     = []
 }
 
-variable "arch" {
-  type        = string
-  description = "Container architecture (amd64 or arm64)"
-  default     = "amd64"
+# advanced
 
+variable "controller_arch" {
+  type        = string
+  description = "Controller node(s) architecture (amd64 or arm64)"
+  default     = "amd64"
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
-    error_message = "The arch must be amd64 or arm64."
+    condition     = contains(["amd64", "arm64"], var.controller_arch)
+    error_message = "The controller_arch must be amd64 or arm64."
+  }
+}
+
+variable "worker_arch" {
+  type        = string
+  description = "Worker node(s) architecture (amd64 or arm64)"
+  default     = "amd64"
+  validation {
+    condition     = contains(["amd64", "arm64"], var.worker_arch)
+    error_message = "The worker_arch must be amd64 or arm64."
   }
 }
 

azure/flatcar-linux/kubernetes/workers.tf

@@ -3,16 +3,19 @@ module "workers" {
   name   = var.cluster_name
 
   # Azure
-  resource_group_name     = azurerm_resource_group.cluster.name
-  region                  = azurerm_resource_group.cluster.location
-  subnet_id               = azurerm_subnet.worker.id
-  security_group_id       = azurerm_network_security_group.worker.id
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  resource_group_name      = azurerm_resource_group.cluster.name
+  location                 = azurerm_resource_group.cluster.location
+  subnet_id                = azurerm_subnet.worker.id
+  security_group_id        = azurerm_network_security_group.worker.id
+  backend_address_pool_ids = local.backend_address_pool_ids
 
-  worker_count = var.worker_count
-  vm_type      = var.worker_type
-  os_image     = var.os_image
-  priority     = var.worker_priority
+  worker_count   = var.worker_count
+  vm_type        = var.worker_type
+  os_image       = var.os_image
+  disk_type      = var.worker_disk_type
+  disk_size      = var.worker_disk_size
+  ephemeral_disk = var.worker_ephemeral_disk
+  priority       = var.worker_priority
 
   # configuration
   kubeconfig            = module.bootstrap.kubeconfig-kubelet
@@ -22,5 +25,5 @@ module "workers" {
   cluster_domain_suffix = var.cluster_domain_suffix
   snippets              = var.worker_snippets
   node_labels           = var.worker_node_labels
-  arch                  = var.arch
+  arch                  = var.worker_arch
 }

azure/flatcar-linux/kubernetes/workers/butane/worker.yaml

@@ -28,7 +28,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin

azure/flatcar-linux/kubernetes/workers/variables.tf

@@ -5,9 +5,9 @@ variable "name" {
 
 # Azure
 
-variable "region" {
+variable "location" {
   type        = string
-  description = "Must be set to the Azure Region of cluster"
+  description = "Must be set to the Azure location of cluster"
 }
 
 variable "resource_group_name" {
@@ -25,9 +25,12 @@ variable "security_group_id" {
   description = "Must be set to the `worker_security_group_id` output by cluster"
 }
 
-variable "backend_address_pool_id" {
-  type        = string
-  description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
+variable "backend_address_pool_ids" {
+  type = object({
+    ipv4 = list(string)
+    ipv6 = list(string)
+  })
+  description = "Must be set to the `backend_address_pool_ids` output by cluster"
 }
 
 # instances
@@ -55,6 +58,24 @@ variable "os_image" {
   }
 }
 
+variable "disk_type" {
+  type        = string
+  description = "Type of managed disk"
+  default     = "Standard_LRS"
+}
+
+variable "disk_size" {
+  type        = number
+  description = "Size of the managed disk in GB"
+  default     = 30
+}
+
+variable "ephemeral_disk" {
+  type        = bool
+  description = "Use ephemeral local disk instead of managed disk (requires vm_type with local storage)"
+  default     = false
+}
+
 variable "priority" {
   type        = string
   description = "Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."

azure/flatcar-linux/kubernetes/workers/workers.tf

@@ -8,25 +8,28 @@ locals {
 }
 
 # Workers scale set
-resource "azurerm_linux_virtual_machine_scale_set" "workers" {
-  resource_group_name = var.resource_group_name
-
-  name      = "${var.name}-worker"
-  location  = var.region
-  sku       = var.vm_type
-  instances = var.worker_count
-  # instance name prefix for instances in the set
-  computer_name_prefix   = "${var.name}-worker"
-  single_placement_group = false
-  custom_data            = base64encode(data.ct_config.worker.rendered)
-  boot_diagnostics {
-    # defaults to a managed storage account
-  }
+resource "azurerm_orchestrated_virtual_machine_scale_set" "workers" {
+  name                        = "${var.name}-worker"
+  resource_group_name         = var.resource_group_name
+  location                    = var.location
+  platform_fault_domain_count = 1
+  sku_name                    = var.vm_type
+  instances                   = var.worker_count
 
   # storage
+  encryption_at_host_enabled = true
   os_disk {
-    storage_account_type = "Standard_LRS"
-    caching              = "ReadWrite"
+    storage_account_type = var.disk_type
+    disk_size_gb         = var.disk_size
+    caching              = "ReadOnly"
+    # Optionally, use the ephemeral disk of the instance type (support varies)
+    dynamic "diff_disk_settings" {
+      for_each = var.ephemeral_disk ? [1] : []
+      content {
+        option    = "Local"
+        placement = "ResourceDisk"
+      }
+    }
   }
 
   # Flatcar Container Linux
@@ -46,13 +49,6 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
     }
   }
 
-  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
-  admin_username = "core"
-  admin_ssh_key {
-    username   = "core"
-    public_key = local.azure_authorized_key
-  }
-
   # network
   network_interface {
     name                      = "nic0"
@@ -60,17 +56,41 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
     network_security_group_id = var.security_group_id
 
     ip_configuration {
-      name      = "ip0"
+      name      = "ipv4"
+      version   = "IPv4"
       primary   = true
       subnet_id = var.subnet_id
-
       # backend address pool to which the NIC should be added
-      load_balancer_backend_address_pool_ids = [var.backend_address_pool_id]
+      load_balancer_backend_address_pool_ids = var.backend_address_pool_ids.ipv4
+    }
+    ip_configuration {
+      name      = "ipv6"
+      version   = "IPv6"
+      subnet_id = var.subnet_id
+      # backend address pool to which the NIC should be added
+      load_balancer_backend_address_pool_ids = var.backend_address_pool_ids.ipv6
+    }
+  }
+
+  # boot
+  user_data_base64 = base64encode(data.ct_config.worker.rendered)
+  boot_diagnostics {
+    # defaults to a managed storage account
+  }
+
+  # Azure requires an RSA admin_ssh_key
+  os_profile {
+    linux_configuration {
+      admin_username = "core"
+      admin_ssh_key {
+        username   = "core"
+        public_key = local.azure_authorized_key
+      }
+      computer_name_prefix = "${var.name}-worker"
     }
   }
 
   # lifecycle
-  upgrade_mode = "Manual"
   # eviction policy may only be set when priority is Spot
   priority        = var.priority
   eviction_policy = var.priority == "Spot" ? "Delete" : null
@@ -79,28 +99,6 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
   }
 }
 
-# Scale up or down to maintain desired number, tolerating deallocations.
-resource "azurerm_monitor_autoscale_setting" "workers" {
-  resource_group_name = var.resource_group_name
-
-  name     = "${var.name}-maintain-desired"
-  location = var.region
-
-  # autoscale
-  enabled            = true
-  target_resource_id = azurerm_linux_virtual_machine_scale_set.workers.id
-
-  profile {
-    name = "default"
-
-    capacity {
-      minimum = var.worker_count
-      default = var.worker_count
-      maximum = var.worker_count
-    }
-  }
-}
-
 # Flatcar Linux worker
 data "ct_config" "worker" {
   content = templatefile("${path.module}/butane/worker.yaml", {

bare-metal/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/fedora-coreos/kubernetes/butane/controller.yaml

@@ -53,7 +53,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -113,7 +113,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \

bare-metal/fedora-coreos/kubernetes/worker/butane/worker.yaml

@@ -25,7 +25,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin

bare-metal/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]

bare-metal/flatcar-linux/kubernetes/butane/controller.yaml

@@ -64,7 +64,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -114,7 +114,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

bare-metal/flatcar-linux/kubernetes/profiles.tf

@@ -60,6 +60,7 @@ data "ct_config" "install" {
     baseurl_flag = var.cached_install ? "-b ${var.matchbox_http_endpoint}/assets/flatcar" : ""
   })
   strict = true
+  snippets = lookup(var.install_snippets, var.controllers.*.name[count.index], [])
 }
 
 # Match each controller by MAC

bare-metal/flatcar-linux/kubernetes/variables.tf

@@ -61,6 +61,12 @@ variable "snippets" {
   default     = {}
 }
 
+variable "install_snippets" {
+  type        = map(list(string))
+  description = "Map from machine names to lists of Container Linux Config snippets to run during install phase"
+  default     = {}
+}
+
 variable "worker_node_labels" {
   type        = map(list(string))
   description = "Map from worker names to lists of initial node labels"

bare-metal/flatcar-linux/kubernetes/worker/butane/worker.yaml

@@ -36,7 +36,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin

bare-metal/flatcar-linux/kubernetes/worker/matchbox.tf

@@ -55,6 +55,7 @@ data "ct_config" "install" {
     baseurl_flag = var.cached_install ? "-b ${var.matchbox_http_endpoint}/assets/flatcar" : ""
   })
   strict = true
+  snippets = var.install_snippets
 }
 
 # Match a worker to a profile by MAC

bare-metal/flatcar-linux/kubernetes/worker/variables.tf

@@ -60,6 +60,12 @@ variable "snippets" {
   default     = []
 }
 
+variable "install_snippets" {
+  type        = list(string)
+  description = "List of Butane snippets to run with the install command"
+  default     = []
+}
+
 variable "node_labels" {
   type        = list(string)
   description = "List of initial node labels"

bare-metal/flatcar-linux/kubernetes/workers.tf

@@ -22,6 +22,7 @@ module "workers" {
   node_labels           = lookup(var.worker_node_labels, var.workers[count.index].name, [])
   node_taints           = lookup(var.worker_node_taints, var.workers[count.index].name, [])
   snippets              = lookup(var.snippets, var.workers[count.index].name, [])
+  install_snippets      = lookup(var.install_snippets, var.workers[count.index].name, [])
 
   # optional
   download_protocol = var.download_protocol

digital-ocean/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/fedora-coreos/kubernetes/butane/controller.yaml

@@ -55,7 +55,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -123,7 +123,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.1
+            quay.io/poseidon/kubelet:v1.30.3
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

digital-ocean/fedora-coreos/kubernetes/butane/worker.yaml

@@ -28,7 +28,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests

digital-ocean/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]

digital-ocean/flatcar-linux/kubernetes/butane/controller.yaml

@@ -66,7 +66,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -117,7 +117,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \

digital-ocean/flatcar-linux/kubernetes/butane/worker.yaml

@@ -38,7 +38,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests

docs/addons/ingress.md

@@ -37,7 +37,7 @@ resource "google_dns_record_set" "some-application" {
 
 ## Azure
 
-On Azure, a load balancer distributes traffic across a backend address pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
+On Azure, an Azure Load Balancer distributes IPv4/IPv6 traffic across backend address pools of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, and namespace.
 
@@ -53,10 +53,10 @@ app2.example.com -> 11.22.33.44
 app3.example.com -> 11.22.33.44
 ```
 
-Find the load balancer's IPv4 address with the Azure console or use the Typhoon module's output `ingress_static_ipv4`. For example, you might use Terraform to manage a Google Cloud DNS record:
+Find the load balancer's addresses with the Azure console or use the Typhoon module's outputs `ingress_static_ipv4` or `ingress_static_ipv6`. For example, you might use Terraform to manage a Google Cloud DNS record:
 
 ```tf
-resource "google_dns_record_set" "some-application" {
+resource "google_dns_record_set" "app-record-a" {
   # DNS zone name
   managed_zone = "example-zone"
 
@@ -66,6 +66,17 @@ resource "google_dns_record_set" "some-application" {
   ttl     = 300
   rrdatas = [module.ramius.ingress_static_ipv4]
 }
+
+resource "google_dns_record_set" "app-record-aaaa" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "AAAA"
+  ttl     = 300
+  rrdatas = [module.ramius.ingress_static_ipv6]
+}
 ```
 
 ## Bare-Metal

docs/advanced/arm64.md

@@ -15,7 +15,7 @@ Create a cluster on AWS with ARM64 controller and worker nodes. Container worklo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.3"
 
       # AWS
       cluster_name = "gravitas"
@@ -40,7 +40,7 @@ Create a cluster on AWS with ARM64 controller and worker nodes. Container worklo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.3"
 
       # AWS
       cluster_name = "gravitas"
@@ -66,9 +66,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes. For Flatcar Linux, describe
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-21-119   Ready    <none>   77s   v1.30.1   10.0.21.119   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
-ip-10-0-32-166   Ready    <none>   80s   v1.30.1   10.0.32.166   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
-ip-10-0-5-79     Ready    <none>   77s   v1.30.1   10.0.5.79     <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
+ip-10-0-21-119   Ready    <none>   77s   v1.30.3   10.0.21.119   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
+ip-10-0-32-166   Ready    <none>   80s   v1.30.3   10.0.32.166   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
+ip-10-0-5-79     Ready    <none>   77s   v1.30.3   10.0.5.79     <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
 ```
 
 ## Hybrid
@@ -79,7 +79,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.3"
 
       # AWS
       cluster_name = "gravitas"
@@ -102,7 +102,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.3"
 
       # AWS
       cluster_name = "gravitas"
@@ -125,7 +125,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.30.3"
 
       # AWS
       vpc_id          = module.gravitas.vpc_id
@@ -149,7 +149,7 @@ Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [wo
 
     ```tf
     module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.30.3"
 
       # AWS
       vpc_id          = module.gravitas.vpc_id
@@ -174,10 +174,10 @@ Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
 ```
 $ kubectl get nodes -o wide
 NAME                       STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                             KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-1-73               Ready    <none>   111m   v1.30.1   10.0.1.73     <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
-ip-10-0-22-79...           Ready    <none>   111m   v1.30.1   10.0.22.79    <none>        Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)   5.10.84-flatcar           containerd://1.5.8
-ip-10-0-24-130             Ready    <none>   111m   v1.30.1   10.0.24.130   <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
-ip-10-0-39-19              Ready    <none>   111m   v1.30.1   10.0.39.19    <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
+ip-10-0-1-73               Ready    <none>   111m   v1.30.3   10.0.1.73     <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
+ip-10-0-22-79...           Ready    <none>   111m   v1.30.3   10.0.22.79    <none>        Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)   5.10.84-flatcar           containerd://1.5.8
+ip-10-0-24-130             Ready    <none>   111m   v1.30.3   10.0.24.130   <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
+ip-10-0-39-19              Ready    <none>   111m   v1.30.3   10.0.39.19    <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
 ```
 
 ## Azure
@@ -186,11 +186,11 @@ Create a cluster on Azure with ARM64 controller and worker nodes. Container work
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.30.3"
 
   # Azure
   cluster_name   = "ramius"
-  region         = "centralus"
+  location       = "centralus"
   dns_zone       = "azure.example.com"
   dns_zone_group = "example-group"
 
@@ -202,6 +202,5 @@ module "ramius" {
   controller_type = "Standard_D2pls_v5"
   worker_type     = "Standard_D2pls_v5"
   worker_count    = 2
-  host_cidr       = "10.0.0.0/20"
 }
 ```

docs/advanced/nodes.md

@@ -36,7 +36,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to
 
     ```tf
     module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
 
       # Google Cloud
       cluster_name  = "yavin"
@@ -57,7 +57,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to
 
     ```tf
     module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.3"
 
       # Google Cloud
       cluster_name = "yavin"
@@ -89,7 +89,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and
 
     ```tf
     module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
 
       # Google Cloud
       cluster_name  = "yavin"
@@ -110,7 +110,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and
 
     ```tf
     module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.3"
 
       # Google Cloud
       cluster_name = "yavin"

docs/advanced/worker-pools.md

@@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.30.3"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.30.3"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -111,14 +111,14 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.30.3"
 
       # Azure
-      region                  = module.ramius.region
-      resource_group_name     = module.ramius.resource_group_name
-      subnet_id               = module.ramius.subnet_id
-      security_group_id       = module.ramius.security_group_id
-      backend_address_pool_id = module.ramius.backend_address_pool_id
+      location                 = module.ramius.location
+      resource_group_name      = module.ramius.resource_group_name
+      subnet_id                = module.ramius.subnet_id
+      security_group_id        = module.ramius.security_group_id
+      backend_address_pool_ids = module.ramius.backend_address_pool_ids
 
       # configuration
       name               = "ramius-spot"
@@ -127,7 +127,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
       # optional
       worker_count = 2
-      vm_type      = "Standard_F4"
+      vm_type      = "Standard_D2as_v5"
       priority     = "Spot"
       os_image     = "/subscriptions/some/path/Microsoft.Compute/images/fedora-coreos-31.20200323.3.2"
     }
@@ -137,14 +137,14 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.30.3"
 
       # Azure
-      region                  = module.ramius.region
-      resource_group_name     = module.ramius.resource_group_name
-      subnet_id               = module.ramius.subnet_id
-      security_group_id       = module.ramius.security_group_id
-      backend_address_pool_id = module.ramius.backend_address_pool_id
+      location                 = module.ramius.location
+      resource_group_name      = module.ramius.resource_group_name
+      subnet_id                = module.ramius.subnet_id
+      security_group_id        = module.ramius.security_group_id
+      backend_address_pool_ids = module.ramius.backend_address_pool_ids
 
       # configuration
       name               = "ramius-spot"
@@ -153,7 +153,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
       # optional
       worker_count = 2
-      vm_type      = "Standard_F4"
+      vm_type      = "Standard_D2as_v5"
       priority     = "Spot"
       os_image     = "flatcar-beta"
     }
@@ -180,7 +180,7 @@ The Azure internal `workers` module supports a number of [variables](https://git
 | resource_group_name | Must be set to `resource_group_name` output by cluster | module.cluster.resource_group_name |
 | subnet_id | Must be set to `subnet_id` output by cluster | module.cluster.subnet_id |
 | security_group_id | Must be set to `security_group_id` output by cluster | module.cluster.security_group_id |
-| backend_address_pool_id | Must be set to `backend_address_pool_id` output by cluster | module.cluster.backend_address_pool_id |
+| backend_address_pool_ids | Must be set to `backend_address_pool_ids` output by cluster | module.cluster.backend_address_pool_ids |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | module.cluster.kubeconfig |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
@@ -207,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.3"
 
       # Google Cloud
       region       = "europe-west2"
@@ -231,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.30.1"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.30.3"
 
       # Google Cloud
       region       = "europe-west2"
@@ -262,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.30.1
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.30.1
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.30.1
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.30.1
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.30.1
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.30.3
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.30.3
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.30.3
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.30.3
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.30.3
 ```
 
 ### Variables

docs/architecture/azure.md

@@ -10,9 +10,9 @@ A load balancer distributes IPv4 TCP/6443 traffic across a backend address pool
 
 ### HTTP/HTTPS Ingress
 
-A load balancer distributes IPv4 TCP/80 and TCP/443 traffic across a backend address pool of workers with a healthy Ingress controller.
+An Azure Load Balancer distributes IPv4/IPv6 TCP/80 and TCP/443 traffic across backend address pools of workers with a healthy Ingress controller.
 
-The Azure LB IPv4 address is output as `ingress_static_ipv4` for use in DNS A records. See [Ingress on Azure](/addons/ingress/#azure).
+The load balancer addresses are output as `ingress_static_ipv4` and `ingress_static_ipv6` for use in DNS A and AAAA records. See [Ingress on Azure](/addons/ingress/#azure).
 
 ### TCP/UDP Services
 
@@ -21,27 +21,25 @@ Load balance TCP/UDP applications by adding rules to the Azure LB (output). A ru
 ```tf
 # Forward traffic to the worker backend address pool
 resource "azurerm_lb_rule" "some-app-tcp" {
-  resource_group_name = module.ramius.resource_group_name
-
   name                           = "some-app-tcp"
+  resource_group_name            = module.ramius.resource_group_name
   loadbalancer_id                = module.ramius.loadbalancer_id
-  frontend_ip_configuration_name = "ingress"
+  frontend_ip_configuration_name = "ingress-ipv4"
 
-  protocol                = "Tcp"
-  frontend_port           = 3333
-  backend_port            = 30333
-  backend_address_pool_id = module.ramius.backend_address_pool_id
-  probe_id                = azurerm_lb_probe.some-app.id
+  protocol                 = "Tcp"
+  frontend_port            = 3333
+  backend_port             = 30333
+  backend_address_pool_ids = module.ramius.backend_address_pool_ids.ipv4
+  probe_id                 = azurerm_lb_probe.some-app.id
 }
 
 # Health check some-app
 resource "azurerm_lb_probe" "some-app" {
+  name                = "some-app"
   resource_group_name = module.ramius.resource_group_name
-
-  name            = "some-app"
-  loadbalancer_id = module.ramius.loadbalancer_id
-  protocol        = "Tcp"
-  port            = 30333
+  loadbalancer_id     = module.ramius.loadbalancer_id
+  protocol            = "Tcp"
+  port                = 30333
 }
 ```
 
@@ -51,9 +49,8 @@ Add firewall rules to the worker security group.
 
 ```tf
 resource "azurerm_network_security_rule" "some-app" {
-  resource_group_name = module.ramius.resource_group_name
-
   name                         = "some-app"
+  resource_group_name          = module.ramius.resource_group_name
   network_security_group_name  = module.ramius.worker_security_group_name
   priority                     = "3001"
   access                       = "Allow"
@@ -62,7 +59,7 @@ resource "azurerm_network_security_rule" "some-app" {
   source_port_range            = "*"
   destination_port_range       = "30333"
   source_address_prefix        = "*"
-  destination_address_prefixes = module.ramius.worker_address_prefixes
+  destination_address_prefixes = module.ramius.worker_address_prefixes.ipv4
 }
 ```
 
@@ -72,6 +69,6 @@ Azure does not provide public IPv6 addresses at the standard SKU.
 
 | IPv6 Feature            | Supported |
 |-------------------------|-----------|
-| Node IPv6 address       | No        |
-| Node Outbound IPv6      | No        |
-| Kubernetes Ingress IPv6 | No        |
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Yes       |

docs/fedora-coreos/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on AWS with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.30.1
-ip-10-0-26-65  Ready   <none>   10m  v1.30.1
-ip-10-0-41-21  Ready   <none>   10m  v1.30.1
+ip-10-0-3-155  Ready   <none>   10m  v1.30.3
+ip-10-0-26-65  Ready   <none>   10m  v1.30.3
+ip-10-0-41-21  Ready   <none>   10m  v1.30.3
 ```
 
 List the pods.

docs/fedora-coreos/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Azure with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -67,15 +67,15 @@ Fedora CoreOS publishes images for Azure, but does not yet upload them. Azure al
 [Download](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) a Fedora CoreOS Azure VHD image, decompress it, and upload it to an Azure storage account container (i.e. bucket) via the UI (quite slow).
 
 ```
-xz -d fedora-coreos-36.20220716.3.1-azure.x86_64.vhd.xz
+xz -d fedora-coreos-40.20240616.3.0-azure.x86_64.vhd.xz
 ```
 
 Create an Azure disk (note disk ID) and create an Azure image from it (note image ID).
 
 ```
-az disk create --name fedora-coreos-36.20220716.3.1 -g GROUP --source https://BUCKET.blob.core.windows.net/fedora-coreos/fedora-coreos-36.20220716.3.1-azure.x86_64.vhd
+az disk create --name fedora-coreos-40.20240616.3.0 -g GROUP --source https://BUCKET.blob.core.windows.net/images/fedora-coreos-40.20240616.3.0-azure.x86_64.vhd
 
-az image create --name fedora-coreos-36.20220716.3.1 -g GROUP --os-type=linux --source /subscriptions/some/path/providers/Microsoft.Compute/disks/fedora-coreos-36.20220716.3.1
+az image create --name fedora-coreos-40.20240616.3.0 -g GROUP --os-type linux --source /subscriptions/some/path/Microsoft.Compute/disks/fedora-coreos-40.20240616.3.0
 ```
 
 Set the [os_image](#variables) in the next step.
@@ -86,11 +86,11 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # Azure
   cluster_name   = "ramius"
-  region         = "centralus"
+  location       = "centralus"
   dns_zone       = "azure.example.com"
   dns_zone_group = "example-group"
 
@@ -100,7 +100,9 @@ module "ramius" {
 
   # optional
   worker_count    = 2
-  host_cidr       = "10.0.0.0/20"
+  network_cidr       = {
+    ipv4 = ["10.0.0.0/20"]
+  }
 }
 ```
 
@@ -161,9 +163,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.30.1
-ramius-worker-000001  Ready   <none>  25m  v1.30.1
-ramius-worker-000002  Ready   <none>  24m  v1.30.1
+ramius-controller-0   Ready   <none>  24m  v1.30.3
+ramius-worker-000001  Ready   <none>  25m  v1.30.3
+ramius-worker-000002  Ready   <none>  24m  v1.30.3
 ```
 
 List the pods.
@@ -197,14 +199,14 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/f
 | Name | Description | Example |
 |:-----|:------------|:--------|
 | cluster_name | Unique cluster name (prepended to dns_zone) | "ramius" |
-| region | Azure region | "centralus" |
+| location | Azure location | "centralus" |
 | dns_zone | Azure DNS zone | "azure.example.com" |
 | dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
 | os_image | Fedora CoreOS image for instances | "/subscriptions/..../custom-image" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-ed25519 AAAAB3NZ..." |
 
 !!! tip
-    Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
+    Locations are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
 
 #### DNS Zone
 
@@ -246,7 +248,7 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | controller_snippets | Controller Butane snippets | [] | [example](/advanced/customization/#usage) |
 | worker_snippets | Worker Butane snippets | [] | [example](/advanced/customization/#usage) |
 | networking | Choice of networking provider | "cilium" | "calico" or "cilium" or "flannel" |
-| host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
+| network_cidr | Virtual network CIDR ranges | { ipv4 = ["10.0.0.0/16"], ipv6 = [ULA, ...] } | { ipv4 = ["10.0.0.0/20"] } |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | worker_node_labels | List of initial worker node labels | [] | ["worker-pool=default"] |

docs/fedora-coreos/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.30.1 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.30.3 cluster on bare-metal with Fedora CoreOS.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -191,7 +191,7 @@ Workers with similar features can be defined inline using the `workers` field as
 
 ```tf
 module "mercury-node1" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.30.3"
 
   # bare-metal
   cluster_name = "mercury"
@@ -313,9 +313,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.30.1
-node2.example.com   Ready   <none>  10m  v1.30.1
-node3.example.com   Ready   <none>  10m  v1.30.1
+node1.example.com   Ready   <none>  10m  v1.30.3
+node2.example.com   Ready   <none>  10m  v1.30.3
+node3.example.com   Ready   <none>  10m  v1.30.3
 ```
 
 List the pods.

docs/fedora-coreos/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on DigitalOcean with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.30.1
-10.132.115.81      Ready   <none>  10m  v1.30.1
-10.132.124.107     Ready   <none>  10m  v1.30.1
+10.132.110.130     Ready   <none>  10m  v1.30.3
+10.132.115.81      Ready   <none>  10m  v1.30.3
+10.132.124.107     Ready   <none>  10m  v1.30.3
 ```
 
 List the pods.

docs/fedora-coreos/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Google Compute Engine with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -73,7 +73,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-coreos/kuberne
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
 ```
 
 List the pods.

docs/flatcar-linux/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on AWS with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -72,7 +72,7 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.3"
 
   # AWS
   cluster_name = "tempest"
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.30.1
-ip-10-0-26-65  Ready   <none>  10m  v1.30.1
-ip-10-0-41-21  Ready   <none>  10m  v1.30.1
+ip-10-0-3-155  Ready   <none>  10m  v1.30.3
+ip-10-0-26-65  Ready   <none>  10m  v1.30.3
+ip-10-0-41-21  Ready   <none>  10m  v1.30.3
 ```
 
 List the pods.

docs/flatcar-linux/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Azure with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -75,11 +75,11 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.30.3"
 
   # Azure
   cluster_name   = "ramius"
-  region         = "centralus"
+  location       = "centralus"
   dns_zone       = "azure.example.com"
   dns_zone_group = "example-group"
 
@@ -88,7 +88,9 @@ module "ramius" {
 
   # optional
   worker_count    = 2
-  host_cidr       = "10.0.0.0/20"
+  network_cidr    = {
+    ipv4 = ["10.0.0.0/20"]
+  }
 }
 ```
 
@@ -149,9 +151,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.30.1
-ramius-worker-000001  Ready   <none>  25m  v1.30.1
-ramius-worker-000002  Ready   <none>  24m  v1.30.1
+ramius-controller-0   Ready   <none>  24m  v1.30.3
+ramius-worker-000001  Ready   <none>  25m  v1.30.3
+ramius-worker-000002  Ready   <none>  24m  v1.30.3
 ```
 
 List the pods.
@@ -185,13 +187,13 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/azure/f
 | Name | Description | Example |
 |:-----|:------------|:--------|
 | cluster_name | Unique cluster name (prepended to dns_zone) | "ramius" |
-| region | Azure region | "centralus" |
+| location | Azure location | "centralus" |
 | dns_zone | Azure DNS zone | "azure.example.com" |
 | dns_zone_group | Resource group where the Azure DNS zone resides | "global" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
 
 !!! tip
-    Regions are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
+    Locations are shown in [docs](https://azure.microsoft.com/en-us/global-infrastructure/regions/) or with `az account list-locations --output table`.
 
 #### DNS Zone
 
@@ -234,7 +236,7 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | networking | Choice of networking provider | "cilium" | "calico" or "cilium" or "flannel" |
-| host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
+| network_cidr | Virtual network CIDR ranges | { ipv4 = ["10.0.0.0/16"], ipv6 = [ULA, ...] } | { ipv4 = ["10.0.0.0/20"] } |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | worker_node_labels | List of initial worker node labels | [] | ["worker-pool=default"] |

docs/flatcar-linux/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.30.1 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.30.3 cluster on bare-metal with Flatcar Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.30.3"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -194,7 +194,7 @@ Workers with similar features can be defined inline using the `workers` field as
 
 ```tf
 module "mercury-node1" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.30.3"
 
   # bare-metal
   cluster_name = "mercury"
@@ -323,9 +323,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.30.1
-node2.example.com   Ready   <none>  10m  v1.30.1
-node3.example.com   Ready   <none>  10m  v1.30.1
+node1.example.com   Ready   <none>  10m  v1.30.3
+node2.example.com   Ready   <none>  10m  v1.30.3
+node3.example.com   Ready   <none>  10m  v1.30.3
 ```
 
 List the pods.

docs/flatcar-linux/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on DigitalOcean with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -81,7 +81,7 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.30.3"
 
   # Digital Ocean
   cluster_name = "nemo"
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.30.1
-10.132.115.81      Ready   <none>  10m  v1.30.1
-10.132.124.107     Ready   <none>  10m  v1.30.1
+10.132.110.130     Ready   <none>  10m  v1.30.3
+10.132.115.81      Ready   <none>  10m  v1.30.3
+10.132.124.107     Ready   <none>  10m  v1.30.3
 ```
 
 List the pods.

docs/flatcar-linux/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.30.1 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Google Compute Engine with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -73,7 +73,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.30.3"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
 ```
 
 List the pods.

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@@ -70,7 +70,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -108,9 +108,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.1
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.1
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.1
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
 ```
 
 List the pods.

docs/topics/maintenance.md

@@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar
 
 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
   ...
 }
 
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.30.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.30.3"
   ...
 }
 ```
@@ -192,7 +192,7 @@ Applying edits to most worker fields will start an instance refresh:
 However, changing `os_stream`/`os_channel` or new AMIs becoming available will NOT change the launch configuration or trigger an Instance Refresh. This allows Fedora CoreOS or Flatcar Linux to auto-update themselves via reboots and avoids unexpected terraform diffs for new AMIs.
 
 !!! note
-    Before Typhoon v1.30.1, worker nodes only used new launch configurations when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
+    Before Typhoon v1.30.3, worker nodes only used new launch configurations when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
 
 ### Google Cloud
 
@@ -233,7 +233,7 @@ Applying edits to most worker fields will start an instance refresh:
 However, changing `os_stream`/`os_channel` or new compute images becoming available will NOT change the launch template or update instances. This allows Fedora CoreOS or Flatcar Linux to auto-update themselves via reboots and avoids unexpected terraform diffs for new AMIs.
 
 !!! note
-    Before Typhoon v1.30.1, worker nodes only used new launch templates when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
+    Before Typhoon v1.30.3, worker nodes only used new launch templates when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
 
 ## Upgrade poseidon/ct
 

google-cloud/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.1 (upstream)
+* Kubernetes v1.30.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

google-cloud/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=e1b1e0c75e77e042cf369f463f0e656297a201a8"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]

google-cloud/fedora-coreos/kubernetes/butane/controller.yaml

@@ -54,7 +54,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.1
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -111,7 +111,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.1
+            quay.io/poseidon/kubelet:v1.30.3
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:

google-cloud/fedora-coreos/kubernetes/controllers.tf

@@ -43,7 +43,7 @@ resource "google_compute_instance" "controllers" {
 
     initialize_params {
       image = data.google_compute_image.fedora-coreos.self_link
-      size  = var.disk_size
+      size  = var.controller_disk_size
     }
   }
 

google-cloud/fedora-coreos/kubernetes/variables.tf

@@ -22,30 +22,6 @@ variable "dns_zone_name" {
 
 # instances
 
-variable "controller_count" {
-  type        = number
-  description = "Number of controllers (i.e. masters)"
-  default     = 1
-}
-
-variable "worker_count" {
-  type        = number
-  description = "Number of workers"
-  default     = 1
-}
-
-variable "controller_type" {
-  type        = string
-  description = "Machine type for controllers (see `gcloud compute machine-types list`)"
-  default     = "n1-standard-1"
-}
-
-variable "worker_type" {
-  type        = string
-  description = "Machine type for controllers (see `gcloud compute machine-types list`)"
-  default     = "n1-standard-1"
-}
-
 variable "os_stream" {
   type        = string
   description = "Fedora CoreOS stream for compute instances (e.g. stable, testing, next)"
@@ -57,7 +33,37 @@ variable "os_stream" {
   }
 }
 
-variable "disk_size" {
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "Machine type for controllers (see `gcloud compute machine-types list`)"
+  default     = "n1-standard-1"
+}
+
+variable "controller_disk_size" {
+  type        = number
+  description = "Size of the disk in GB"
+  default     = 30
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "worker_type" {
+  type        = string
+  description = "Machine type for controllers (see `gcloud compute machine-types list`)"
+  default     = "n1-standard-1"
+}
+
+variable "worker_disk_size" {
   type        = number
   description = "Size of the disk in GB"
   default     = 30