google: Use regional instance templates for workers

* Use regional instance templates for the worker node regional
managed instance groups. Regional instance templates are kept in
the associated region, whereas the older "global" instance templates
were kept in a particular region (regardless of where the MIG region)
so outages in a region X could affect clusters in a region Y which
is undesired

CHANGES.md

@@ -4,11 +4,31 @@ Notable changes between versions.
 
 ## Latest
 
-### Azure
+## v1.31.0
 
-* Allow controller and worker nodes to use different CPU architectures
+* Kubernetes [v1.31.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.31.md#v1310)
-  * Add `controller_arch` and `worker_arch` variables
+* Use Cilium kube-proxy replacement mode when `cilium` networking is chosen ([#1501](https://github.com/poseidon/typhoon/pull/1501))
-  * Remove the `arch` variable
+* Fix invalid flannel-cni container image for those using `flannel` networking ([#1497](https://github.com/poseidon/typhoon/pull/1497))
+
+### AWS
+
+* Use EC2 resource-based hostnames instead of IP-based hostnames ([#1499](https://github.com/poseidon/typhoon/pull/1499))
+  * The Amazon DNS server can resolve A and AAAA queries to IPv4 and IPv6 node addresses
+* Tag controller node EBS volumes with a name based on the controller node name
+
+### Google
+
+* Use `google_compute_region_instance_template` instead of `google_compute_instance_template`
+  * Google's regional instance template metadata is kept in the associated region for greater resiliency. The "global" instance templates were kept in a single region
+
+## v1.30.4
+
+* Kubernetes [v1.30.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#v1304)
+* Update Cilium from v1.15.7 to [v1.16.1](https://github.com/cilium/cilium/releases/tag/v1.16.1)
+* Update CoreDNS from v1.11.1 to v1.11.3
+* Remove `enable_aggregation` variable for Kubernetes Aggregation Layer, always set to true
+* Remove `cluster_domain_suffix` variable, always use "cluster.local"
+* Remove `enable_reporting` variable for analytics, always set to false
 
 ## v1.30.3
 
@@ -18,12 +38,12 @@ Notable changes between versions.
 
 ### AWS
 
-* Allow configuring controller and worker disks ([#1482](https://github.com/poseidon/typhoon/pull/1482))
+* Configure controller and worker disks ([#1482](https://github.com/poseidon/typhoon/pull/1482))
   * Add `controller_disk_type`, `controller_disk_size`, and `controller_disk_iops` variables
   * Add `worker_disk_type`, `worker_disk_size`, and `worker_disk_iops` variables
   * Remove `disk_type`, `disk_size`, and `disk_iops` variables
   * Fix propagating settings to worker disks, previously ignored
-* Allow configuring CPU pricing model for burstable instance types ([#1482](https://github.com/poseidon/typhoon/pull/1482))
+* Configure CPU pricing model for burstable instance types ([#1482](https://github.com/poseidon/typhoon/pull/1482))
   * Add `controller_cpu_credits` and `worker_cpu_credits` variables (`standard` or `unlimited`)
 * Configure controller or worker instance architecture ([#1485](https://github.com/poseidon/typhoon/pull/1485))
   * Add `controller_arch` and `worker_arch` variables (`amd64` or `arm64`)

README.md

@@ -18,7 +18,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
@@ -78,7 +78,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -117,9 +117,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.31.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.31.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.31.0
 ```
 
 List the pods.
@@ -127,9 +127,10 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
-kube-system   calico-node-1cs8z                         2/2    Running   0         6m
+kube-system   cilium-1cs8z                              1/1    Running   0         6m
-kube-system   calico-node-d1l5b                         2/2    Running   0         6m
+kube-system   cilium-d1l5b                              1/1    Running   0         6m
-kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   cilium-sp9ps                              1/1    Running   0         6m
+kube-system   cilium-operator-68d778b448-g744f          1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   kube-apiserver-controller-0               1/1    Running   0         6m

addons/cilium/config.tf

@@ -128,8 +128,8 @@ resource "kubernetes_config_map" "cilium" {
     enable-bpf-masquerade = "true"
 
     # kube-proxy
-    kube-proxy-replacement                      = "false"
+    kube-proxy-replacement                      = "true"
-    kube-proxy-replacement-healthz-bind-address = ""
+    kube-proxy-replacement-healthz-bind-address = ":10256"
     enable-session-affinity                     = "true"
 
     # ClusterIPs from host namespace

addons/cilium/daemonset.tf

@@ -61,7 +61,7 @@ resource "kubernetes_daemonset" "cilium" {
         # https://github.com/cilium/cilium/pull/24075
         init_container {
           name    = "install-cni"
-          image   = "quay.io/cilium/cilium:v1.16.0"
+          image   = "quay.io/cilium/cilium:v1.16.1"
           command = ["/install-plugin.sh"]
           security_context {
             allow_privilege_escalation = true
@@ -80,7 +80,7 @@ resource "kubernetes_daemonset" "cilium" {
         # We use nsenter command with host's cgroup and mount namespaces enabled.
         init_container {
           name  = "mount-cgroup"
-          image = "quay.io/cilium/cilium:v1.16.0"
+          image = "quay.io/cilium/cilium:v1.16.1"
           command = [
             "sh",
             "-ec",
@@ -115,7 +115,7 @@ resource "kubernetes_daemonset" "cilium" {
 
         init_container {
           name    = "clean-cilium-state"
-          image   = "quay.io/cilium/cilium:v1.16.0"
+          image   = "quay.io/cilium/cilium:v1.16.1"
           command = ["/init-container.sh"]
           security_context {
             allow_privilege_escalation = true
@@ -139,7 +139,7 @@ resource "kubernetes_daemonset" "cilium" {
 
         container {
           name    = "cilium-agent"
-          image   = "quay.io/cilium/cilium:v1.16.0"
+          image   = "quay.io/cilium/cilium:v1.16.1"
           command = ["cilium-agent"]
           args = [
             "--config-dir=/tmp/cilium/config-map"

addons/cilium/deployment.tf

@@ -58,7 +58,7 @@ resource "kubernetes_deployment" "operator" {
         enable_service_links            = false
         container {
           name    = "cilium-operator"
-          image   = "quay.io/cilium/operator-generic:v1.16.0"
+          image   = "quay.io/cilium/operator-generic:v1.16.1"
           command = ["cilium-operator-generic"]
           args = [
             "--config-dir=/tmp/cilium/config-map",

aws/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -9,9 +9,6 @@ module "bootstrap" {
   network_mtu           = var.network_mtu
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
   components            = var.components
 }

aws/fedora-coreos/kubernetes/butane/controller.yaml

@@ -57,7 +57,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -116,7 +116,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.3
+            quay.io/poseidon/kubelet:v1.31.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -149,7 +149,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

aws/fedora-coreos/kubernetes/controllers.tf

@@ -20,10 +20,8 @@ resource "aws_instance" "controllers" {
   tags = {
     Name = "${var.cluster_name}-controller-${count.index}"
   }
-
   instance_type = var.controller_type
   ami           = var.controller_arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
-  user_data     = data.ct_config.controllers.*.rendered[count.index]
 
   # storage
   root_block_device {
@@ -31,7 +29,9 @@ resource "aws_instance" "controllers" {
     volume_size = var.controller_disk_size
     iops        = var.controller_disk_iops
     encrypted   = true
-    tags        = {}
+    tags = {
+      Name = "${var.cluster_name}-controller-${count.index}"
+    }
   }
 
   # network
@@ -39,6 +39,10 @@ resource "aws_instance" "controllers" {
   subnet_id                   = element(aws_subnet.public.*.id, count.index)
   vpc_security_group_ids      = [aws_security_group.controller.id]
 
+  # boot
+  user_data = data.ct_config.controllers.*.rendered[count.index]
+
+  # cost
   credit_specification {
     cpu_credits = var.controller_cpu_credits
   }
@@ -65,7 +69,6 @@ data "ct_config" "controllers" {
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

aws/fedora-coreos/kubernetes/network.tf

@@ -47,17 +47,25 @@ resource "aws_route" "egress-ipv6" {
 resource "aws_subnet" "public" {
   count = length(data.aws_availability_zones.all.names)
 
-  vpc_id            = aws_vpc.network.id
-  availability_zone = data.aws_availability_zones.all.names[count.index]
-
-  cidr_block                      = cidrsubnet(var.host_cidr, 4, count.index)
-  ipv6_cidr_block                 = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
-  map_public_ip_on_launch         = true
-  assign_ipv6_address_on_creation = true
-
   tags = {
     "Name" = "${var.cluster_name}-public-${count.index}"
   }
+  vpc_id            = aws_vpc.network.id
+  availability_zone = data.aws_availability_zones.all.names[count.index]
+
+  # IPv4 and IPv6 CIDR blocks
+  cidr_block      = cidrsubnet(var.host_cidr, 4, count.index)
+  ipv6_cidr_block = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
+
+  # Assign IPv4 and IPv6 addresses to instances
+  map_public_ip_on_launch         = true
+  assign_ipv6_address_on_creation = true
+
+  # Hostnames assigned to instances
+  # resource-name: <ec2-instance-id>.region.compute.internal
+  private_dns_hostname_type_on_launch            = "resource-name"
+  enable_resource_name_dns_a_record_on_launch    = true
+  enable_resource_name_dns_aaaa_record_on_launch = true
 }
 
 resource "aws_route_table_association" "public" {

aws/fedora-coreos/kubernetes/variables.tf

@@ -164,32 +164,12 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
-  default     = "cluster.local"
-}
-
 # advanced
 
 variable "controller_arch" {

aws/fedora-coreos/kubernetes/workers.tf

@@ -6,23 +6,24 @@ module "workers" {
   vpc_id          = aws_vpc.network.id
   subnet_ids      = aws_subnet.public.*.id
   security_groups = [aws_security_group.worker.id]
-  worker_count    = var.worker_count
+
-  instance_type   = var.worker_type
+  # instances
-  os_stream       = var.os_stream
+  os_stream     = var.os_stream
-  arch            = var.worker_arch
+  worker_count  = var.worker_count
-  disk_type       = var.worker_disk_type
+  instance_type = var.worker_type
-  disk_size       = var.worker_disk_size
+  arch          = var.worker_arch
-  disk_iops       = var.worker_disk_iops
+  disk_type     = var.worker_disk_type
-  cpu_credits     = var.worker_cpu_credits
+  disk_size     = var.worker_disk_size
-  spot_price      = var.worker_price
+  disk_iops     = var.worker_disk_iops
-  target_groups   = var.worker_target_groups
+  cpu_credits   = var.worker_cpu_credits
+  spot_price    = var.worker_price
+  target_groups = var.worker_target_groups
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig         = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key = var.ssh_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets           = var.worker_snippets
-  snippets              = var.worker_snippets
+  node_labels        = var.worker_node_labels
-  node_labels           = var.worker_node_labels
 }
 

aws/fedora-coreos/kubernetes/workers/butane/worker.yaml

@@ -29,7 +29,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -104,7 +104,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

aws/fedora-coreos/kubernetes/workers/variables.tf

@@ -108,12 +108,6 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
-
 variable "node_labels" {
   type        = list(string)
   description = "List of initial node labels"
@@ -126,15 +120,14 @@ variable "node_taints" {
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
 
 variable "arch" {
   type        = string
   description = "Container architecture (amd64 or arm64)"
   default     = "amd64"
-
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
+    condition     = contains(["amd64", "arm64"], var.arch)
     error_message = "The arch must be amd64 or arm64."
   }
 }

aws/fedora-coreos/kubernetes/workers/workers.tf

@@ -3,16 +3,14 @@ resource "aws_autoscaling_group" "workers" {
   name = "${var.name}-worker"
 
   # count
-  desired_capacity          = var.worker_count
+  desired_capacity = var.worker_count
-  min_size                  = var.worker_count
+  min_size         = var.worker_count
-  max_size                  = var.worker_count + 2
+  max_size         = var.worker_count + 2
-  default_cooldown          = 30
-  health_check_grace_period = 30
 
   # network
   vpc_zone_identifier = var.subnet_ids
 
-  # template
+  # instance template
   launch_template {
     id      = aws_launch_template.worker.id
     version = aws_launch_template.worker.latest_version
@@ -32,6 +30,11 @@ resource "aws_autoscaling_group" "workers" {
       min_healthy_percentage = 90
     }
   }
+  # Grace period before checking new instance's health
+  health_check_grace_period = 30
+  # Cooldown period between scaling activities
+  default_cooldown = 30
+
 
   lifecycle {
     # override the default destroy and replace update behavior
@@ -56,11 +59,6 @@ resource "aws_launch_template" "worker" {
   name_prefix   = "${var.name}-worker"
   image_id      = local.ami_id
   instance_type = var.instance_type
-  monitoring {
-    enabled = false
-  }
-
-  user_data = sensitive(base64encode(data.ct_config.worker.rendered))
 
   # storage
   ebs_optimized = true
@@ -76,14 +74,26 @@ resource "aws_launch_template" "worker" {
   }
 
   # network
-  vpc_security_group_ids = var.security_groups
+  network_interfaces {
+    associate_public_ip_address = true
+    security_groups             = var.security_groups
+  }
+
+  # boot
+  user_data = sensitive(base64encode(data.ct_config.worker.rendered))
 
   # metadata
   metadata_options {
     http_tokens = "optional"
   }
+  monitoring {
+    enabled = false
+  }
 
-  # spot
+  # cost
+  credit_specification {
+    cpu_credits = var.cpu_credits
+  }
   dynamic "instance_market_options" {
     for_each = var.spot_price > 0 ? [1] : []
     content {
@@ -94,10 +104,6 @@ resource "aws_launch_template" "worker" {
     }
   }
 
-  credit_specification {
-    cpu_credits = var.cpu_credits
-  }
-
   lifecycle {
     // Override the default destroy and replace update behavior
     create_before_destroy = true
@@ -111,7 +117,6 @@ data "ct_config" "worker" {
     kubeconfig             = indent(10, var.kubeconfig)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
   })

aws/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

aws/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -9,9 +9,6 @@ module "bootstrap" {
   network_mtu           = var.network_mtu
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
   components            = var.components
 }

aws/flatcar-linux/kubernetes/butane/controller.yaml

@@ -58,7 +58,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -109,7 +109,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \
@@ -148,7 +148,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

aws/flatcar-linux/kubernetes/controllers.tf

@@ -20,11 +20,8 @@ resource "aws_instance" "controllers" {
   tags = {
     Name = "${var.cluster_name}-controller-${count.index}"
   }
-
   instance_type = var.controller_type
-
+  ami           = local.ami_id
-  ami       = local.ami_id
-  user_data = data.ct_config.controllers.*.rendered[count.index]
 
   # storage
   root_block_device {
@@ -32,7 +29,9 @@ resource "aws_instance" "controllers" {
     volume_size = var.controller_disk_size
     iops        = var.controller_disk_iops
     encrypted   = true
-    tags        = {}
+    tags = {
+      Name = "${var.cluster_name}-controller-${count.index}"
+    }
   }
 
   # network
@@ -40,6 +39,10 @@ resource "aws_instance" "controllers" {
   subnet_id                   = element(aws_subnet.public.*.id, count.index)
   vpc_security_group_ids      = [aws_security_group.controller.id]
 
+  # boot
+  user_data = data.ct_config.controllers.*.rendered[count.index]
+
+  # cost
   credit_specification {
     cpu_credits = var.controller_cpu_credits
   }
@@ -66,7 +69,6 @@ data "ct_config" "controllers" {
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

aws/flatcar-linux/kubernetes/network.tf

@@ -47,17 +47,25 @@ resource "aws_route" "egress-ipv6" {
 resource "aws_subnet" "public" {
   count = length(data.aws_availability_zones.all.names)
 
-  vpc_id            = aws_vpc.network.id
-  availability_zone = data.aws_availability_zones.all.names[count.index]
-
-  cidr_block                      = cidrsubnet(var.host_cidr, 4, count.index)
-  ipv6_cidr_block                 = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
-  map_public_ip_on_launch         = true
-  assign_ipv6_address_on_creation = true
-
   tags = {
     "Name" = "${var.cluster_name}-public-${count.index}"
   }
+  vpc_id            = aws_vpc.network.id
+  availability_zone = data.aws_availability_zones.all.names[count.index]
+
+  # IPv4 and IPv6 CIDR blocks
+  cidr_block      = cidrsubnet(var.host_cidr, 4, count.index)
+  ipv6_cidr_block = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
+
+  # Assign IPv4 and IPv6 addresses to instances
+  map_public_ip_on_launch         = true
+  assign_ipv6_address_on_creation = true
+
+  # Hostnames assigned to instances
+  # resource-name: <ec2-instance-id>.region.compute.internal
+  private_dns_hostname_type_on_launch            = "resource-name"
+  enable_resource_name_dns_a_record_on_launch    = true
+  enable_resource_name_dns_aaaa_record_on_launch = true
 }
 
 resource "aws_route_table_association" "public" {

aws/flatcar-linux/kubernetes/variables.tf

@@ -164,31 +164,13 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
-  default     = "cluster.local"
-}
 
 variable "controller_arch" {
   type        = string
@@ -210,7 +192,6 @@ variable "worker_arch" {
   }
 }
 
-
 variable "daemonset_tolerations" {
   type        = list(string)
   description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"

aws/flatcar-linux/kubernetes/workers.tf

@@ -6,22 +6,23 @@ module "workers" {
   vpc_id          = aws_vpc.network.id
   subnet_ids      = aws_subnet.public.*.id
   security_groups = [aws_security_group.worker.id]
-  worker_count    = var.worker_count
+
-  instance_type   = var.worker_type
+  # instances
-  os_image        = var.os_image
+  os_image      = var.os_image
-  arch            = var.worker_arch
+  worker_count  = var.worker_count
-  disk_type       = var.worker_disk_type
+  instance_type = var.worker_type
-  disk_size       = var.worker_disk_size
+  arch          = var.worker_arch
-  disk_iops       = var.worker_disk_iops
+  disk_type     = var.worker_disk_type
-  spot_price      = var.worker_price
+  disk_size     = var.worker_disk_size
-  target_groups   = var.worker_target_groups
+  disk_iops     = var.worker_disk_iops
+  spot_price    = var.worker_price
+  target_groups = var.worker_target_groups
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig         = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key = var.ssh_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets           = var.worker_snippets
-  snippets              = var.worker_snippets
+  node_labels        = var.worker_node_labels
-  node_labels           = var.worker_node_labels
 }
 

aws/flatcar-linux/kubernetes/workers/butane/worker.yaml

@@ -30,7 +30,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -103,7 +103,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

aws/flatcar-linux/kubernetes/workers/variables.tf

@@ -108,12 +108,6 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
-
 variable "node_labels" {
   type        = list(string)
   description = "List of initial node labels"
@@ -134,7 +128,7 @@ variable "arch" {
   default     = "amd64"
 
   validation {
-    condition     = var.arch == "amd64" || var.arch == "arm64"
+    condition     = contains(["amd64", "arm64"], var.arch)
     error_message = "The arch must be amd64 or arm64."
   }
 }

aws/flatcar-linux/kubernetes/workers/workers.tf

@@ -3,16 +3,14 @@ resource "aws_autoscaling_group" "workers" {
   name = "${var.name}-worker"
 
   # count
-  desired_capacity          = var.worker_count
+  desired_capacity = var.worker_count
-  min_size                  = var.worker_count
+  min_size         = var.worker_count
-  max_size                  = var.worker_count + 2
+  max_size         = var.worker_count + 2
-  default_cooldown          = 30
-  health_check_grace_period = 30
 
   # network
   vpc_zone_identifier = var.subnet_ids
 
-  # template
+  # instance template
   launch_template {
     id      = aws_launch_template.worker.id
     version = aws_launch_template.worker.latest_version
@@ -32,6 +30,10 @@ resource "aws_autoscaling_group" "workers" {
       min_healthy_percentage = 90
     }
   }
+  # Grace period before checking new instance's health
+  health_check_grace_period = 30
+  # Cooldown period between scaling activities
+  default_cooldown = 30
 
   lifecycle {
     # override the default destroy and replace update behavior
@@ -56,11 +58,6 @@ resource "aws_launch_template" "worker" {
   name_prefix   = "${var.name}-worker"
   image_id      = local.ami_id
   instance_type = var.instance_type
-  monitoring {
-    enabled = false
-  }
-
-  user_data = sensitive(base64encode(data.ct_config.worker.rendered))
 
   # storage
   ebs_optimized = true
@@ -76,14 +73,26 @@ resource "aws_launch_template" "worker" {
   }
 
   # network
-  vpc_security_group_ids = var.security_groups
+  network_interfaces {
+    associate_public_ip_address = true
+    security_groups             = var.security_groups
+  }
+
+  # boot
+  user_data = sensitive(base64encode(data.ct_config.worker.rendered))
 
   # metadata
   metadata_options {
     http_tokens = "optional"
   }
+  monitoring {
+    enabled = false
+  }
 
-  # spot
+  # cost
+  credit_specification {
+    cpu_credits = var.cpu_credits
+  }
   dynamic "instance_market_options" {
     for_each = var.spot_price > 0 ? [1] : []
     content {
@@ -94,10 +103,6 @@ resource "aws_launch_template" "worker" {
     }
   }
 
-  credit_specification {
-    cpu_credits = var.cpu_credits
-  }
-
   lifecycle {
     // Override the default destroy and replace update behavior
     create_before_destroy = true
@@ -111,7 +116,6 @@ data "ct_config" "worker" {
     kubeconfig             = indent(10, var.kubeconfig)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
   })

azure/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -14,9 +14,6 @@ module "bootstrap" {
 
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
   components            = var.components
 }

azure/fedora-coreos/kubernetes/butane/controller.yaml

@@ -54,7 +54,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -111,7 +111,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.3
+            quay.io/poseidon/kubelet:v1.31.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -144,7 +144,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

azure/fedora-coreos/kubernetes/controllers.tf

@@ -163,7 +163,6 @@ data "ct_config" "controllers" {
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

azure/fedora-coreos/kubernetes/variables.tf

@@ -27,7 +27,6 @@ variable "os_image" {
   description = "Fedora CoreOS image for instances"
 }
 
-
 variable "controller_count" {
   type        = number
   description = "Number of controllers (i.e. masters)"
@@ -145,31 +144,13 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
 
 variable "daemonset_tolerations" {
   type        = list(string)

azure/fedora-coreos/kubernetes/versions.tf

@@ -3,7 +3,7 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    azurerm = ">= 2.8, < 4.0"
+    azurerm = ">= 2.8"
     null    = ">= 2.1"
     ct = {
       source  = "poseidon/ct"

azure/fedora-coreos/kubernetes/workers.tf

@@ -9,20 +9,20 @@ module "workers" {
   security_group_id        = azurerm_network_security_group.worker.id
   backend_address_pool_ids = local.backend_address_pool_ids
 
+  # instances
+  os_image       = var.os_image
   worker_count   = var.worker_count
   vm_type        = var.worker_type
-  os_image       = var.os_image
   disk_type      = var.worker_disk_type
   disk_size      = var.worker_disk_size
   ephemeral_disk = var.worker_ephemeral_disk
   priority       = var.worker_priority
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig           = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key   = var.ssh_authorized_key
-  azure_authorized_key  = var.azure_authorized_key
+  azure_authorized_key = var.azure_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr         = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets             = var.worker_snippets
-  snippets              = var.worker_snippets
+  node_labels          = var.worker_node_labels
-  node_labels           = var.worker_node_labels
 }

azure/fedora-coreos/kubernetes/workers/butane/worker.yaml

@@ -26,7 +26,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -99,7 +99,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

azure/fedora-coreos/kubernetes/workers/variables.tf

@@ -120,12 +120,3 @@ variable "node_taints" {
   description = "List of initial node taints"
   default     = []
 }
-
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = string
-  default     = "cluster.local"
-}
-

azure/fedora-coreos/kubernetes/workers/versions.tf

@@ -3,7 +3,7 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    azurerm = ">= 2.8, < 4.0"
+    azurerm = ">= 2.8"
     ct = {
       source  = "poseidon/ct"
       version = "~> 0.13"

azure/fedora-coreos/kubernetes/workers/workers.tf

@@ -84,7 +84,6 @@ data "ct_config" "worker" {
     kubeconfig             = indent(10, var.kubeconfig)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
   })

azure/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/flatcar-linux/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

azure/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -14,9 +14,6 @@ module "bootstrap" {
 
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
   components            = var.components
 }

azure/flatcar-linux/kubernetes/butane/controller.yaml

@@ -56,7 +56,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -105,7 +105,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \
@@ -144,7 +144,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

azure/flatcar-linux/kubernetes/controllers.tf

@@ -185,7 +185,6 @@ data "ct_config" "controllers" {
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

azure/flatcar-linux/kubernetes/variables.tf

@@ -150,18 +150,6 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
@@ -196,14 +184,6 @@ variable "daemonset_tolerations" {
   default     = []
 }
 
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
-
 variable "components" {
   description = "Configure pre-installed cluster components"
   # Component configs are passed through to terraform-render-bootstrap,

azure/flatcar-linux/kubernetes/versions.tf

@@ -3,7 +3,7 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    azurerm = ">= 2.8, < 4.0"
+    azurerm = ">= 2.8"
     null    = ">= 2.1"
     ct = {
       source  = "poseidon/ct"

azure/flatcar-linux/kubernetes/workers.tf

@@ -18,12 +18,11 @@ module "workers" {
   priority       = var.worker_priority
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig           = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key   = var.ssh_authorized_key
-  azure_authorized_key  = var.azure_authorized_key
+  azure_authorized_key = var.azure_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr         = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets             = var.worker_snippets
-  snippets              = var.worker_snippets
+  node_labels          = var.worker_node_labels
-  node_labels           = var.worker_node_labels
+  arch                 = var.worker_arch
-  arch                  = var.worker_arch
 }

azure/flatcar-linux/kubernetes/workers/butane/worker.yaml

@@ -28,7 +28,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -99,7 +99,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

azure/flatcar-linux/kubernetes/workers/variables.tf

@@ -137,12 +137,3 @@ variable "arch" {
     error_message = "The arch must be amd64 or arm64."
   }
 }
-
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = string
-  default     = "cluster.local"
-}
-

azure/flatcar-linux/kubernetes/workers/versions.tf

@@ -3,7 +3,7 @@
 terraform {
   required_version = ">= 0.13.0, < 2.0.0"
   required_providers {
-    azurerm = ">= 2.8, < 4.0"
+    azurerm = ">= 2.8"
     ct = {
       source  = "poseidon/ct"
       version = "~> 0.13"

azure/flatcar-linux/kubernetes/workers/workers.tf

@@ -105,7 +105,6 @@ data "ct_config" "worker" {
     kubeconfig             = indent(10, var.kubeconfig)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
   })

bare-metal/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]
@@ -10,9 +10,6 @@ module "bootstrap" {
   network_ip_autodetection_method = var.network_ip_autodetection_method
   pod_cidr                        = var.pod_cidr
   service_cidr                    = var.service_cidr
-  cluster_domain_suffix           = var.cluster_domain_suffix
-  enable_reporting                = var.enable_reporting
-  enable_aggregation              = var.enable_aggregation
   components                      = var.components
 }
 

bare-metal/fedora-coreos/kubernetes/butane/controller.yaml

@@ -53,7 +53,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -113,7 +113,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=-/usr/bin/podman rm bootstrap
         ExecStart=/usr/bin/podman run --name bootstrap \
             --network host \
@@ -154,7 +154,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

bare-metal/fedora-coreos/kubernetes/profiles.tf

@@ -59,7 +59,6 @@ data "ct_config" "controllers" {
     etcd_name              = var.controllers.*.name[count.index]
     etcd_initial_cluster   = join(",", formatlist("%s=https://%s:2380", var.controllers.*.name, var.controllers.*.domain))
     cluster_dns_service_ip = module.bootstrap.cluster_dns_service_ip
-    cluster_domain_suffix  = var.cluster_domain_suffix
     ssh_authorized_key     = var.ssh_authorized_key
   })
   strict   = true

bare-metal/fedora-coreos/kubernetes/variables.tf

@@ -139,25 +139,7 @@ variable "kernel_args" {
   default     = []
 }
 
-variable "enable_reporting" {
+# advanced
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = string
-  default     = "cluster.local"
-}
 
 variable "components" {
   description = "Configure pre-installed cluster components"

bare-metal/fedora-coreos/kubernetes/worker/butane/worker.yaml

@@ -25,7 +25,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -108,7 +108,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

bare-metal/fedora-coreos/kubernetes/worker/matchbox.tf

@@ -53,7 +53,6 @@ data "ct_config" "worker" {
     domain_name            = var.domain
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
   })

bare-metal/fedora-coreos/kubernetes/worker/variables.tf

@@ -103,9 +103,3 @@ The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for
 EOD
   default     = "10.3.0.0/16"
 }
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = string
-  default     = "cluster.local"
-}

bare-metal/fedora-coreos/kubernetes/workers.tf

@@ -15,13 +15,12 @@ module "workers" {
   domain = var.workers[count.index].domain
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig         = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key = var.ssh_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  node_labels        = lookup(var.worker_node_labels, var.workers[count.index].name, [])
-  node_labels           = lookup(var.worker_node_labels, var.workers[count.index].name, [])
+  node_taints        = lookup(var.worker_node_taints, var.workers[count.index].name, [])
-  node_taints           = lookup(var.worker_node_taints, var.workers[count.index].name, [])
+  snippets           = lookup(var.snippets, var.workers[count.index].name, [])
-  snippets              = lookup(var.snippets, var.workers[count.index].name, [])
 
   # optional
   cached_install = var.cached_install

bare-metal/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

bare-metal/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name                    = var.cluster_name
   api_servers                     = [var.k8s_domain_name]
@@ -10,9 +10,6 @@ module "bootstrap" {
   network_ip_autodetection_method = var.network_ip_autodetection_method
   pod_cidr                        = var.pod_cidr
   service_cidr                    = var.service_cidr
-  cluster_domain_suffix           = var.cluster_domain_suffix
-  enable_reporting                = var.enable_reporting
-  enable_aggregation              = var.enable_aggregation
   components                      = var.components
 }
 

bare-metal/flatcar-linux/kubernetes/butane/controller.yaml

@@ -64,7 +64,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -114,7 +114,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \
@@ -155,7 +155,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

bare-metal/flatcar-linux/kubernetes/profiles.tf

@@ -88,7 +88,6 @@ data "ct_config" "controllers" {
     etcd_name              = var.controllers.*.name[count.index]
     etcd_initial_cluster   = join(",", formatlist("%s=https://%s:2380", var.controllers.*.name, var.controllers.*.domain))
     cluster_dns_service_ip = module.bootstrap.cluster_dns_service_ip
-    cluster_domain_suffix  = var.cluster_domain_suffix
     ssh_authorized_key     = var.ssh_authorized_key
   })
   strict   = true

bare-metal/flatcar-linux/kubernetes/variables.tf

@@ -144,18 +144,6 @@ variable "kernel_args" {
   default     = []
 }
 
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "oem_type" {
   type        = string
   description = <<EOD
@@ -167,13 +155,7 @@ EOD
   default     = ""
 }
 
-# unofficial, undocumented, unsupported
+# advanced
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
 
 variable "components" {
   description = "Configure pre-installed cluster components"

bare-metal/flatcar-linux/kubernetes/worker/butane/worker.yaml

@@ -36,7 +36,7 @@ systemd:
         After=docker.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -113,7 +113,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

bare-metal/flatcar-linux/kubernetes/worker/matchbox.tf

@@ -79,7 +79,6 @@ data "ct_config" "worker" {
     domain_name            = var.domain
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
     node_labels            = join(",", var.node_labels)
     node_taints            = join(",", var.node_taints)
   })

bare-metal/flatcar-linux/kubernetes/worker/variables.tf

@@ -114,13 +114,3 @@ The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for
 EOD
   default     = "10.3.0.0/16"
 }
-
-
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
-
-

bare-metal/flatcar-linux/kubernetes/workers.tf

@@ -15,13 +15,12 @@ module "workers" {
   domain = var.workers[count.index].domain
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig         = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key = var.ssh_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  node_labels        = lookup(var.worker_node_labels, var.workers[count.index].name, [])
-  node_labels           = lookup(var.worker_node_labels, var.workers[count.index].name, [])
+  node_taints        = lookup(var.worker_node_taints, var.workers[count.index].name, [])
-  node_taints           = lookup(var.worker_node_taints, var.workers[count.index].name, [])
+  snippets           = lookup(var.snippets, var.workers[count.index].name, [])
-  snippets              = lookup(var.snippets, var.workers[count.index].name, [])
 
   # optional
   download_protocol = var.download_protocol

digital-ocean/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -11,11 +11,8 @@ module "bootstrap" {
   network_encapsulation = "vxlan"
   network_mtu           = "1450"
 
-  pod_cidr              = var.pod_cidr
+  pod_cidr     = var.pod_cidr
-  service_cidr          = var.service_cidr
+  service_cidr = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  components   = var.components
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
-  components            = var.components
 }
 

digital-ocean/fedora-coreos/kubernetes/butane/controller.yaml

@@ -55,7 +55,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -123,7 +123,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.3
+            quay.io/poseidon/kubelet:v1.31.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -151,7 +151,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

digital-ocean/fedora-coreos/kubernetes/butane/worker.yaml

@@ -28,7 +28,7 @@ systemd:
         After=afterburn.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/afterburn
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -104,7 +104,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

digital-ocean/fedora-coreos/kubernetes/controllers.tf

@@ -74,7 +74,6 @@ data "ct_config" "controllers" {
       for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
     ])
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

digital-ocean/fedora-coreos/kubernetes/variables.tf

@@ -86,25 +86,7 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
+# advanced
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
 
 variable "components" {
   description = "Configure pre-installed cluster components"

digital-ocean/fedora-coreos/kubernetes/workers.tf

@@ -62,7 +62,6 @@ resource "digitalocean_tag" "workers" {
 data "ct_config" "worker" {
   content = templatefile("${path.module}/butane/worker.yaml", {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.worker_snippets

digital-ocean/flatcar-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

digital-ocean/flatcar-linux/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name = var.cluster_name
   api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -11,11 +11,8 @@ module "bootstrap" {
   network_encapsulation = "vxlan"
   network_mtu           = "1450"
 
-  pod_cidr              = var.pod_cidr
+  pod_cidr     = var.pod_cidr
-  service_cidr          = var.service_cidr
+  service_cidr = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  components   = var.components
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
-  components            = var.components
 }
 

digital-ocean/flatcar-linux/kubernetes/butane/controller.yaml

@@ -66,7 +66,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -117,7 +117,7 @@ systemd:
         Type=oneshot
         RemainAfterExit=true
         WorkingDirectory=/opt/bootstrap
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStart=/usr/bin/docker run \
             -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
             -v /opt/bootstrap/assets:/assets:ro \
@@ -153,7 +153,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

digital-ocean/flatcar-linux/kubernetes/butane/worker.yaml

@@ -38,7 +38,7 @@ systemd:
         After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         EnvironmentFile=/run/metadata/coreos
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -103,7 +103,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

digital-ocean/flatcar-linux/kubernetes/controllers.tf

@@ -79,7 +79,6 @@ data "ct_config" "controllers" {
       for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
     ])
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

digital-ocean/flatcar-linux/kubernetes/variables.tf

@@ -86,25 +86,7 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-variable "enable_reporting" {
+# advanced
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
-# unofficial, undocumented, unsupported
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
 
 variable "components" {
   description = "Configure pre-installed cluster components"

digital-ocean/flatcar-linux/kubernetes/workers.tf

@@ -60,7 +60,6 @@ resource "digitalocean_tag" "workers" {
 data "ct_config" "worker" {
   content = templatefile("${path.module}/butane/worker.yaml", {
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.worker_snippets

docs/advanced/arm64.md

@@ -1,13 +1,11 @@
 # ARM64
 
-Typhoon supports ARM64 Kubernetes clusters with ARM64 controller and worker nodes (full-cluster) or adding worker pools of ARM64 nodes to clusters with an x86/amd64 control plane for a hybdrid (mixed-arch) cluster.
+Typhoon supports Kubernetes clusters with ARM64 controller or worker nodes on several platforms:
-
-Typhoon ARM64 clusters (full-cluster or mixed-arch) are available on:
 
 * AWS with Fedora CoreOS or Flatcar Linux
 * Azure with Flatcar Linux
 
-## Cluster
+## AWS
 
 Create a cluster on AWS with ARM64 controller and worker nodes. Container workloads must be `arm64` compatible and use `arm64` (or multi-arch) container images.
 
@@ -15,24 +13,23 @@ Create a cluster on AWS with ARM64 controller and worker nodes. Container worklo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.31.0"
 
       # AWS
       cluster_name = "gravitas"
       dns_zone     = "aws.example.com"
       dns_zone_id  = "Z3PAABBCFAKEC0"
 
+      # instances
+      controller_type = "t4g.small"
+      controller_arch = "arm64"
+      worker_count    = 2
+      worker_type     = "t4g.small"
+      worker_arch     = "arm64"
+      worker_price    = "0.0168"
+
       # configuration
       ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-      # optional
-      arch         = "arm64"
-      networking   = "cilium"
-      worker_count = 2
-      worker_price = "0.0168"
-
-      controller_type = "t4g.small"
-      worker_type     = "t4g.small"
     }
     ```
 
@@ -40,24 +37,23 @@ Create a cluster on AWS with ARM64 controller and worker nodes. Container worklo
 
     ```tf
     module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.31.0"
 
       # AWS
       cluster_name = "gravitas"
       dns_zone     = "aws.example.com"
       dns_zone_id  = "Z3PAABBCFAKEC0"
 
+      # instances
+      controller_type = "t4g.small"
+      controller_arch = "arm64"
+      worker_count    = 2
+      worker_type     = "t4g.small"
+      worker_arch     = "arm64"
+      worker_price    = "0.0168"
+
       # configuration
       ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-      # optional
-      arch         = "arm64"
-      networking   = "cilium"
-      worker_count = 2
-      worker_price = "0.0168"
-
-      controller_type = "t4g.small"
-      worker_type     = "t4g.small"
     }
     ```
 
@@ -66,118 +62,9 @@ Verify the cluster has only arm64 (`aarch64`) nodes. For Flatcar Linux, describe
 ```
 $ kubectl get nodes -o wide
 NAME             STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-21-119   Ready    <none>   77s   v1.30.3   10.0.21.119   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
+ip-10-0-21-119   Ready    <none>   77s   v1.31.0   10.0.21.119   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
-ip-10-0-32-166   Ready    <none>   80s   v1.30.3   10.0.32.166   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
+ip-10-0-32-166   Ready    <none>   80s   v1.31.0   10.0.32.166   <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
-ip-10-0-5-79     Ready    <none>   77s   v1.30.3   10.0.5.79     <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
+ip-10-0-5-79     Ready    <none>   77s   v1.31.0   10.0.5.79     <none>        Fedora CoreOS 35.20211215.3.0   5.15.7-200.fc35.aarch64   containerd://1.5.8
-```
-
-## Hybrid
-
-Create a hybrid/mixed arch cluster by defining an AWS cluster. Then define a [worker pool](worker-pools.md#aws) with ARM64 workers. Optional taints are added to aid in scheduling.
-
-=== "FCOS Cluster"
-
-    ```tf
-    module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.3"
-
-      # AWS
-      cluster_name = "gravitas"
-      dns_zone     = "aws.example.com"
-      dns_zone_id  = "Z3PAABBCFAKEC0"
-
-      # configuration
-      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-      # optional
-      networking   = "cilium"
-      worker_count = 2
-      worker_price = "0.021"
-
-      daemonset_tolerations = ["arch"]     # important
-    }
-    ```
-
-=== "Flatcar Cluster"
-
-    ```tf
-    module "gravitas" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.3"
-
-      # AWS
-      cluster_name = "gravitas"
-      dns_zone     = "aws.example.com"
-      dns_zone_id  = "Z3PAABBCFAKEC0"
-
-      # configuration
-      ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-      # optional
-      networking   = "cilium"
-      worker_count = 2
-      worker_price = "0.021"
-
-      daemonset_tolerations = ["arch"]     # important
-    }
-    ```
-
-=== "FCOS ARM64 Workers"
-
-    ```tf
-    module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.30.3"
-
-      # AWS
-      vpc_id          = module.gravitas.vpc_id
-      subnet_ids      = module.gravitas.subnet_ids
-      security_groups = module.gravitas.worker_security_groups
-
-      # configuration
-      name               = "gravitas-arm64"
-      kubeconfig         = module.gravitas.kubeconfig
-      ssh_authorized_key = var.ssh_authorized_key
-
-      # optional
-      arch          = "arm64"
-      instance_type = "t4g.small"
-      spot_price    = "0.0168"
-      node_taints   = ["arch=arm64:NoSchedule"]
-    }
-    ```
-
-=== "Flatcar ARM64 Workers"
-
-    ```tf
-    module "gravitas-arm64" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.30.3"
-
-      # AWS
-      vpc_id          = module.gravitas.vpc_id
-      subnet_ids      = module.gravitas.subnet_ids
-      security_groups = module.gravitas.worker_security_groups
-
-      # configuration
-      name               = "gravitas-arm64"
-      kubeconfig         = module.gravitas.kubeconfig
-      ssh_authorized_key = var.ssh_authorized_key
-
-      # optional
-      arch          = "arm64"
-      instance_type = "t4g.small"
-      spot_price    = "0.0168"
-      node_taints   = ["arch=arm64:NoSchedule"]
-    }
-    ```
-
-Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
-
-```
-$ kubectl get nodes -o wide
-NAME                       STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                             KERNEL-VERSION            CONTAINER-RUNTIME
-ip-10-0-1-73               Ready    <none>   111m   v1.30.3   10.0.1.73     <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
-ip-10-0-22-79...           Ready    <none>   111m   v1.30.3   10.0.22.79    <none>        Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)   5.10.84-flatcar           containerd://1.5.8
-ip-10-0-24-130             Ready    <none>   111m   v1.30.3   10.0.24.130   <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
-ip-10-0-39-19              Ready    <none>   111m   v1.30.3   10.0.39.19    <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
 ```
 
 ## Azure
@@ -186,7 +73,7 @@ Create a cluster on Azure with ARM64 controller and worker nodes. Container work
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.31.0"
 
   # Azure
   cluster_name   = "ramius"
@@ -194,13 +81,128 @@ module "ramius" {
   dns_zone       = "azure.example.com"
   dns_zone_group = "example-group"
 
+  # instances
+  controller_arch = "arm64"
+  controller_type = "Standard_B2pls_v5"
+  worker_count    = 2
+  controller_arch = "arm64"
+  worker_type     = "Standard_D2pls_v5"
+
   # configuration
   ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-
-  # optional
-  arch            = "arm64"
-  controller_type = "Standard_D2pls_v5"
-  worker_type     = "Standard_D2pls_v5"
-  worker_count    = 2
 }
 ```
+
+## Hybrid
+
+Create a hybrid/mixed arch cluster by defining a cluster where [worker pool(s)](worker-pools.md#aws) have a different instance type architecture than controllers or other workers. Taints are added to aid in scheduling.
+
+Here's an AWS example,
+
+=== "FCOS Cluster"
+
+    ```tf
+    module "gravitas" {
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.31.0"
+
+      # AWS
+      cluster_name = "gravitas"
+      dns_zone     = "aws.example.com"
+      dns_zone_id  = "Z3PAABBCFAKEC0"
+
+      # instances
+      worker_count = 2
+      worker_arch  = "arm64"
+      worker_type  = "t4g.medium"
+      worker_price = "0.021"
+
+      # configuration
+      daemonset_tolerations = ["arch"]     # important
+      networking            = "cilium"
+      ssh_authorized_key    = "ssh-ed25519 AAAAB3Nz..."
+    }
+    ```
+
+=== "Flatcar Cluster"
+
+    ```tf
+    module "gravitas" {
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.31.0"
+
+      # AWS
+      cluster_name = "gravitas"
+      dns_zone     = "aws.example.com"
+      dns_zone_id  = "Z3PAABBCFAKEC0"
+
+      # instances
+      worker_count = 2
+      worker_arch  = "arm64"
+      worker_type  = "t4g.medium"
+      worker_price = "0.021"
+
+      # configuration
+      daemonset_tolerations = ["arch"]     # important
+      networking            = "cilium"
+      ssh_authorized_key    = "ssh-ed25519 AAAAB3Nz..."
+    }
+    ```
+
+=== "FCOS ARM64 Workers"
+
+    ```tf
+    module "gravitas-arm64" {
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.31.0"
+
+      # AWS
+      vpc_id          = module.gravitas.vpc_id
+      subnet_ids      = module.gravitas.subnet_ids
+      security_groups = module.gravitas.worker_security_groups
+
+      # instances
+      arch          = "arm64"
+      instance_type = "t4g.small"
+      spot_price    = "0.0168"
+
+      # configuration
+      name               = "gravitas-arm64"
+      kubeconfig         = module.gravitas.kubeconfig
+      node_taints        = ["arch=arm64:NoSchedule"]
+      ssh_authorized_key = var.ssh_authorized_key
+    }
+    ```
+
+=== "Flatcar ARM64 Workers"
+
+    ```tf
+    module "gravitas-arm64" {
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.31.0"
+
+      # AWS
+      vpc_id          = module.gravitas.vpc_id
+      subnet_ids      = module.gravitas.subnet_ids
+      security_groups = module.gravitas.worker_security_groups
+
+      # instances
+      arch          = "arm64"
+      instance_type = "t4g.small"
+      spot_price    = "0.0168"
+
+      # configuration
+      name               = "gravitas-arm64"
+      kubeconfig         = module.gravitas.kubeconfig
+      node_taints        = ["arch=arm64:NoSchedule"]
+      ssh_authorized_key = var.ssh_authorized_key
+    }
+    ```
+
+Verify amd64 (x86_64) and arm64 (aarch64) nodes are present.
+
+```
+$ kubectl get nodes -o wide
+NAME                       STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                             KERNEL-VERSION            CONTAINER-RUNTIME
+ip-10-0-1-73               Ready    <none>   111m   v1.31.0   10.0.1.73     <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
+ip-10-0-22-79...           Ready    <none>   111m   v1.31.0   10.0.22.79    <none>        Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)   5.10.84-flatcar           containerd://1.5.8
+ip-10-0-24-130             Ready    <none>   111m   v1.31.0   10.0.24.130   <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
+ip-10-0-39-19              Ready    <none>   111m   v1.31.0   10.0.39.19    <none>        Fedora CoreOS 35.20211215.3.0                        5.15.7-200.fc35.x86_64    containerd://1.5.8
+```
+

docs/advanced/nodes.md

@@ -36,7 +36,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to
 
     ```tf
     module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
 
       # Google Cloud
       cluster_name  = "yavin"
@@ -57,7 +57,7 @@ Add custom initial worker node labels to default workers or worker pool nodes to
 
     ```tf
     module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.31.0"
 
       # Google Cloud
       cluster_name = "yavin"
@@ -89,7 +89,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and
 
     ```tf
     module "yavin" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
 
       # Google Cloud
       cluster_name  = "yavin"
@@ -110,7 +110,7 @@ Add custom initial taints on worker pool nodes to indicate a node is unique and
 
     ```tf
     module "yavin-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.31.0"
 
       # Google Cloud
       cluster_name = "yavin"

docs/advanced/worker-pools.md

@@ -19,7 +19,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes/workers?ref=v1.31.0"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -42,7 +42,7 @@ Create a cluster following the AWS [tutorial](../flatcar-linux/aws.md#cluster).
 
     ```tf
     module "tempest-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes/workers?ref=v1.31.0"
 
       # AWS
       vpc_id          = module.tempest.vpc_id
@@ -111,7 +111,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes/workers?ref=v1.31.0"
 
       # Azure
       location                 = module.ramius.location
@@ -137,7 +137,7 @@ Create a cluster following the Azure [tutorial](../flatcar-linux/azure.md#cluste
 
     ```tf
     module "ramius-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes/workers?ref=v1.31.0"
 
       # Azure
       location                 = module.ramius.location
@@ -207,7 +207,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes/workers?ref=v1.31.0"
 
       # Google Cloud
       region       = "europe-west2"
@@ -231,7 +231,7 @@ Create a cluster following the Google Cloud [tutorial](../flatcar-linux/google-c
 
     ```tf
     module "yavin-worker-pool" {
-      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.30.3"
+      source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes/workers?ref=v1.31.0"
 
       # Google Cloud
       region       = "europe-west2"
@@ -262,11 +262,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.30.3
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.31.0
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.30.3
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.31.0
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.30.3
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.31.0
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.30.3
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.31.0
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.30.3
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.31.0
 ```
 
 ### Variables

docs/fedora-coreos/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on AWS with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -72,19 +72,19 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # AWS
   cluster_name = "tempest"
   dns_zone     = "aws.example.com"
   dns_zone_id  = "Z3PAABBCFAKEC0"
 
-  # configuration
+  # instances
-  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-  # optional
   worker_count = 2
   worker_type  = "t3.small"
+
+  # configuration
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 }
 ```
 
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.30.3
+ip-10-0-3-155  Ready   <none>   10m  v1.31.0
-ip-10-0-26-65  Ready   <none>   10m  v1.30.3
+ip-10-0-26-65  Ready   <none>   10m  v1.31.0
-ip-10-0-41-21  Ready   <none>   10m  v1.30.3
+ip-10-0-41-21  Ready   <none>   10m  v1.31.0
 ```
 
 List the pods.
@@ -155,9 +155,9 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                   READY  STATUS    RESTARTS  AGE
-kube-system   calico-node-1m5bf                      2/2    Running   0         34m
+kube-system   cilium-1m5bf                           1/1    Running   0         34m
-kube-system   calico-node-7jmr1                      2/2    Running   0         34m
+kube-system   cilium-7jmr1                           1/1    Running   0         34m
-kube-system   calico-node-bknc8                      2/2    Running   0         34m
+kube-system   cilium-bknc8                           1/1    Running   0         34m
 kube-system   coredns-1187388186-wx1lg               1/1    Running   0         34m
 kube-system   coredns-1187388186-qjnvp               1/1    Running   0         34m
 kube-system   kube-apiserver-ip-10-0-3-155           1/1    Running   0         34m
@@ -206,16 +206,21 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
+| os_stream | Fedora CoreOS stream for instances | "stable" | "testing", "next" |
 | controller_count | Number of controllers (i.e. masters) | 1 | 1 |
-| worker_count | Number of workers | 1 | 3 |
 | controller_type | EC2 instance type for controllers | "t3.small" | See below |
+| controller_disk_size | Size of EBS volume in GB | 30 | 100 |
+| controller_disk_type | Type of EBS volume | gp3 | io1 |
+| controller_disk_iops | IOPS of EBS volume | 3000 | 4000 |
+| controller_cpu_credits | Burstable CPU pricing model | null (i.e. auto) | standard, unlimited |
+| worker_count | Number of workers | 1 | 3 |
 | worker_type | EC2 instance type for workers | "t3.small" | See below |
-| os_stream | Fedora CoreOS stream for compute instances | "stable" | "testing", "next" |
+| worker_disk_size | Size of EBS volume in GB | 30 | 100 |
-| disk_size | Size of the EBS volume in GB | 30 | 100 |
+| worker_disk_type | Type of EBS volume | gp3 | io1 |
-| disk_type | Type of the EBS volume | "gp3" | standard, gp2, gp3, io1 |
+| worker_disk_iops | IOPS of EBS volume | 3000 | 4000 |
-| disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
+| worker_cpu_credits | Burstable CPU pricing model | null (i.e. auto) | standard, unlimited |
-| worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | worker_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0 | 0.10 |
+| worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | controller_snippets | Controller Butane snippets | [] | [examples](/advanced/customization/) |
 | worker_snippets | Worker Butane snippets | [] | [examples](/advanced/customization/) |
 | networking | Choice of networking provider | "cilium" | "calico" or "cilium" or "flannel" |
@@ -228,7 +233,7 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/).
 
 !!! warning
-    Do not choose a `controller_type` smaller than `t2.small`. Smaller instances are not sufficient for running a controller.
+    Do not choose a `controller_type` smaller than `t3.small`. Smaller instances are not sufficient for running a controller.
 
 !!! tip "MTU"
     If your EC2 instance type supports [Jumbo frames](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) (most do), we recommend you change the `network_mtu` to 8981! You will get better pod-to-pod bandwidth.

docs/fedora-coreos/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Azure with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on Azure with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -86,23 +86,23 @@ Define a Kubernetes cluster using the module `azure/fedora-coreos/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # Azure
   cluster_name   = "ramius"
   location       = "centralus"
   dns_zone       = "azure.example.com"
   dns_zone_group = "example-group"
-
+  network_cidr   = {
-  # configuration
-  os_image           = "/subscriptions/some/path/Microsoft.Compute/images/fedora-coreos-36.20220716.3.1"
-  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-  # optional
-  worker_count    = 2
-  network_cidr       = {
     ipv4 = ["10.0.0.0/20"]
   }
+
+  # instances
+  os_image     = "/subscriptions/some/path/Microsoft.Compute/images/fedora-coreos-36.20220716.3.1"
+  worker_count = 2
+
+  # configuration
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
 }
 ```
 
@@ -163,9 +163,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.30.3
+ramius-controller-0   Ready   <none>  24m  v1.31.0
-ramius-worker-000001  Ready   <none>  25m  v1.30.3
+ramius-worker-000001  Ready   <none>  25m  v1.31.0
-ramius-worker-000002  Ready   <none>  24m  v1.30.3
+ramius-worker-000002  Ready   <none>  24m  v1.31.0
 ```
 
 List the pods.
@@ -175,9 +175,9 @@ $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                        READY  STATUS    RESTARTS  AGE
 kube-system   coredns-7c6fbb4f4b-b6qzx                    1/1    Running   0         26m
 kube-system   coredns-7c6fbb4f4b-j2k3d                    1/1    Running   0         26m
-kube-system   calico-node-1m5bf                           2/2    Running   0         26m
+kube-system   cilium-1m5bf                                1/1    Running   0         26m
-kube-system   calico-node-7jmr1                           2/2    Running   0         26m
+kube-system   cilium-7jmr1                                1/1    Running   0         26m
-kube-system   calico-node-bknc8                           2/2    Running   0         26m
+kube-system   cilium-bknc8                                1/1    Running   0         26m
 kube-system   kube-apiserver-ramius-controller-0          1/1    Running   0         26m
 kube-system   kube-controller-manager-ramius-controller-0 1/1    Running   0         26m
 kube-system   kube-proxy-j4vpq                            1/1    Running   0         26m
@@ -240,10 +240,14 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
 | controller_count | Number of controllers (i.e. masters) | 1 | 1 |
-| worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "Standard_B2s" | See below |
+| controller_disk_type | Managed disk for controllers | Premium_LRS | Standard_LRS |
+| controller_disk_size | Managed disk size in GB      | 30 | 50 |
+| worker_count | Number of workers | 1 | 3 |
 | worker_type | Machine type for workers | "Standard_D2as_v5" | See below |
-| disk_size | Size of the disk in GB | 30 | 100 |
+| worker_disk_type | Managed disk for workers | Standard_LRS | Premium_LRS |
+| worker_disk_size | Size of the disk in GB | 30 | 100 |
+| worker_ephemeral_disk | Use ephemeral local disk instead of managed disk | false | true |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
 | controller_snippets | Controller Butane snippets | [] | [example](/advanced/customization/#usage) |
 | worker_snippets | Worker Butane snippets | [] | [example](/advanced/customization/#usage) |
@@ -255,9 +259,6 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 
 Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
 
-!!! warning
-    Unlike AWS and GCP, Azure requires its *virtual* networks to have non-overlapping IPv4 CIDRs (yeah, go figure). Instead of each cluster just using `10.0.0.0/16` for instances, each Azure cluster's `host_cidr` must be non-overlapping (e.g. 10.0.0.0/20 for the 1st cluster, 10.0.16.0/20 for the 2nd cluster, etc).
-
 !!! warning
     Do not choose a `controller_type` smaller than `Standard_B2s`. Smaller instances are not sufficient for running a controller.
 

docs/fedora-coreos/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.30.3 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.31.0 cluster on bare-metal with Fedora CoreOS.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -191,7 +191,7 @@ Workers with similar features can be defined inline using the `workers` field as
 
 ```tf
 module "mercury-node1" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.31.0"
 
   # bare-metal
   cluster_name = "mercury"
@@ -313,9 +313,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.30.3
+node1.example.com   Ready   <none>  10m  v1.31.0
-node2.example.com   Ready   <none>  10m  v1.30.3
+node2.example.com   Ready   <none>  10m  v1.31.0
-node3.example.com   Ready   <none>  10m  v1.30.3
+node3.example.com   Ready   <none>  10m  v1.31.0
 ```
 
 List the pods.
@@ -323,9 +323,10 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
-kube-system   calico-node-6qp7f                          2/2       Running   1          11m
+kube-system   cilium-6qp7f                               1/1       Running   1          11m
-kube-system   calico-node-gnjrm                          2/2       Running   0          11m
+kube-system   cilium-gnjrm                               1/1       Running   0          11m
-kube-system   calico-node-llbgt                          2/2       Running   0          11m
+kube-system   cilium-llbgt                               1/1       Running   0          11m
+kube-system   cilium-operator-68d778b448-g744f           1/1       Running   0          11m
 kube-system   coredns-1187388186-dj3pd                   1/1       Running   0          11m
 kube-system   coredns-1187388186-mx9rt                   1/1       Running   0          11m
 kube-system   kube-apiserver-node1.example.com           1/1       Running   0          11m

docs/fedora-coreos/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on DigitalOcean with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on DigitalOcean with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -81,19 +81,19 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # Digital Ocean
   cluster_name = "nemo"
   region       = "nyc3"
   dns_zone     = "digital-ocean.example.com"
 
-  # configuration
+  # instances
-  os_image         = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id
+  os_image     = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id
-  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
-
-  # optional
   worker_count = 2
+
+  # configuration
+  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
 }
 ```
 
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.30.3
+10.132.110.130     Ready   <none>  10m  v1.31.0
-10.132.115.81      Ready   <none>  10m  v1.30.3
+10.132.115.81      Ready   <none>  10m  v1.31.0
-10.132.124.107     Ready   <none>  10m  v1.30.3
+10.132.124.107     Ready   <none>  10m  v1.31.0
 ```
 
 List the pods.
@@ -166,9 +166,9 @@ List the pods.
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
 kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
-kube-system   calico-node-1m5bf                          2/2       Running   0          11m
+kube-system   cilium-1m5bf                               1/1       Running   0          11m
-kube-system   calico-node-7jmr1                          2/2       Running   0          11m
+kube-system   cilium-7jmr1                               1/1       Running   0          11m
-kube-system   calico-node-bknc8                          2/2       Running   0          11m
+kube-system   cilium-bknc8                               1/1       Running   0          11m
 kube-system   kube-apiserver-ip-10.132.115.81            1/1       Running   0          11m
 kube-system   kube-controller-manager-ip-10.132.115.81   1/1       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m

docs/fedora-coreos/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on Google Compute Engine with Fedora CoreOS.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -73,7 +73,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-coreos/kuberne
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -81,11 +81,11 @@ module "yavin" {
   dns_zone      = "example.com"
   dns_zone_name = "example-zone"
 
+  # instances
+  worker_count = 2
+
   # configuration
   ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
-
-  # optional
-  worker_count = 2
 }
 ```
 
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.31.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.31.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.31.0
 ```
 
 List the pods.
@@ -157,9 +157,9 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
-kube-system   calico-node-1cs8z                         2/2    Running   0         6m
+kube-system   cilium-1cs8z                              1/1    Running   0         6m
-kube-system   calico-node-d1l5b                         2/2    Running   0         6m
+kube-system   cilium-d1l5b                              1/1    Running   0         6m
-kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   cilium-sp9ps                              1/1    Running   0         6m
 kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-controller-0               1/1    Running   0         6m
@@ -211,12 +211,13 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
-| controller_count | Number of controllers (i.e. masters) | 1 | 3 |
-| worker_count | Number of workers | 1 | 3 |
-| controller_type | Machine type for controllers | "n1-standard-1" | See below |
-| worker_type | Machine type for workers | "n1-standard-1" | See below |
 | os_stream | Fedora CoreOS stream for compute instances | "stable" | "stable", "testing", "next" |
-| disk_size | Size of the disk in GB | 30 | 100 |
+| controller_count | Number of controllers (i.e. masters) | 1 | 3 |
+| controller_type | Machine type for controllers | "n1-standard-1" | See below |
+| controller_disk_type | Controller disk size in GB | 30 | 20 |
+| worker_count | Number of workers | 1 | 3 |
+| worker_type | Machine type for workers | "n1-standard-1" | See below |
+| worker_disk_size | Worker disk size in GB | 30 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Butane snippets | [] | [examples](/advanced/customization/) |
 | worker_snippets | Worker Butane snippets | [] | [examples](/advanced/customization/) |

docs/flatcar-linux/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on AWS with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on AWS with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -72,19 +72,19 @@ Define a Kubernetes cluster using the module `aws/flatcar-linux/kubernetes`.
 
 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/flatcar-linux/kubernetes?ref=v1.31.0"
 
   # AWS
   cluster_name = "tempest"
   dns_zone     = "aws.example.com"
   dns_zone_id  = "Z3PAABBCFAKEC0"
 
-  # configuration
+  # instances
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-
-  # optional
   worker_count = 2
   worker_type  = "t3.small"
+
+  # configuration
+  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
 }
 ```
 
@@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.30.3
+ip-10-0-3-155  Ready   <none>  10m  v1.31.0
-ip-10-0-26-65  Ready   <none>  10m  v1.30.3
+ip-10-0-26-65  Ready   <none>  10m  v1.31.0
-ip-10-0-41-21  Ready   <none>  10m  v1.30.3
+ip-10-0-41-21  Ready   <none>  10m  v1.31.0
 ```
 
 List the pods.
@@ -155,9 +155,9 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
-kube-system   calico-node-1m5bf                         2/2    Running   0         34m
+kube-system   cilium-1m5bf                              1/1    Running   0         34m
-kube-system   calico-node-7jmr1                         2/2    Running   0         34m
+kube-system   cilium-7jmr1                              1/1    Running   0         34m
-kube-system   calico-node-bknc8                         2/2    Running   0         34m
+kube-system   cilium-bknc8                              1/1    Running   0         34m
 kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m
 kube-system   coredns-1187388186-qjnvp                  1/1    Running   0         34m
 kube-system   kube-apiserver-ip-10-0-3-155              1/1    Running   0         34m
@@ -206,16 +206,19 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
-| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
-| worker_count | Number of workers | 1 | 3 |
-| controller_type | EC2 instance type for controllers | "t3.small" | See below |
-| worker_type | EC2 instance type for workers | "t3.small" | See below |
 | os_image | AMI channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
-| disk_size | Size of the EBS volume in GB | 30 | 100 |
+| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
-| disk_type | Type of the EBS volume | "gp3" | standard, gp2, gp3, io1 |
+| controller_type | EC2 instance type for controllers | "t3.small" | See below |
-| disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
+| controller_disk_size | Size of EBS volume in GB | 30 | 100 |
-| worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
+| controller_disk_type | Type of EBS volume | gp3 | io1 |
+| controller_disk_iops | IOPS of EBS volume | 3000 | 4000 |
+| controller_cpu_credits | Burstable CPU pricing model | null (i.e. auto) | standard, unlimited |
+| worker_disk_size | Size of EBS volume in GB | 30 | 100 |
+| worker_disk_type | Type of EBS volume | gp3 | io1 |
+| worker_disk_iops | IOPS of EBS volume | 3000 | 4000 |
+| worker_cpu_credits | Burstable CPU pricing model | null (i.e. auto) | standard, unlimited |
 | worker_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0/null | 0.10 |
+| worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "cilium" | "calico" or "cilium" or "flannel" |
@@ -228,7 +231,7 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/).
 
 !!! warning
-    Do not choose a `controller_type` smaller than `t2.small`. Smaller instances are not sufficient for running a controller.
+    Do not choose a `controller_type` smaller than `t3.small`. Smaller instances are not sufficient for running a controller.
 
 !!! tip "MTU"
     If your EC2 instance type supports [Jumbo frames](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) (most do), we recommend you change the `network_mtu` to 8981! You will get better pod-to-pod bandwidth.

docs/flatcar-linux/azure.md

@@ -1,6 +1,6 @@
 # Azure
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Azure with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on Azure with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -75,22 +75,22 @@ Define a Kubernetes cluster using the module `azure/flatcar-linux/kubernetes`.
 
 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/flatcar-linux/kubernetes?ref=v1.31.0"
 
   # Azure
   cluster_name   = "ramius"
   location       = "centralus"
   dns_zone       = "azure.example.com"
   dns_zone_group = "example-group"
+  network_cidr   = {
+    ipv4 = ["10.0.0.0/20"]
+  }
+
+  # instances
+  worker_count    = 2
 
   # configuration
   ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-
-  # optional
-  worker_count    = 2
-  network_cidr    = {
-    ipv4 = ["10.0.0.0/20"]
-  }
 }
 ```
 
@@ -151,9 +151,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.30.3
+ramius-controller-0   Ready   <none>  24m  v1.31.0
-ramius-worker-000001  Ready   <none>  25m  v1.30.3
+ramius-worker-000001  Ready   <none>  25m  v1.31.0
-ramius-worker-000002  Ready   <none>  24m  v1.30.3
+ramius-worker-000002  Ready   <none>  24m  v1.31.0
 ```
 
 List the pods.
@@ -163,9 +163,9 @@ $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                        READY  STATUS    RESTARTS  AGE
 kube-system   coredns-7c6fbb4f4b-b6qzx                    1/1    Running   0         26m
 kube-system   coredns-7c6fbb4f4b-j2k3d                    1/1    Running   0         26m
-kube-system   calico-node-1m5bf                           2/2    Running   0         26m
+kube-system   cilium-1m5bf                                1/1    Running   0         26m
-kube-system   calico-node-7jmr1                           2/2    Running   0         26m
+kube-system   cilium-7jmr1                                1/1    Running   0         26m
-kube-system   calico-node-bknc8                           2/2    Running   0         26m
+kube-system   cilium-bknc8                                1/1    Running   0         26m
 kube-system   kube-apiserver-ramius-controller-0          1/1    Running   0         26m
 kube-system   kube-controller-manager-ramius-controller-0 1/1    Running   0         26m
 kube-system   kube-proxy-j4vpq                            1/1    Running   0         26m
@@ -226,12 +226,16 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
-| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
-| worker_count | Number of workers | 1 | 3 |
-| controller_type | Machine type for controllers | "Standard_B2s" | See below |
-| worker_type | Machine type for workers | "Standard_D2as_v5" | See below |
 | os_image | Channel for a Container Linux derivative | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
-| disk_size | Size of the disk in GB | 30 | 100 |
+| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
+| controller_type | Machine type for controllers | "Standard_B2s" | See below |
+| controller_disk_type | Managed disk for controllers | Premium_LRS | Standard_LRS |
+| controller_disk_size | Managed disk size in GB      | 30 | 50 |
+| worker_count | Number of workers | 1 | 3 |
+| worker_type | Machine type for workers | "Standard_D2as_v5" | See below |
+| worker_disk_type | Managed disk for workers | Standard_LRS | Premium_LRS |
+| worker_disk_size | Size of the disk in GB | 30 | 100 |
+| worker_ephemeral_disk | Use ephemeral local disk instead of managed disk | false | true |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
@@ -243,9 +247,6 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 
 Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
 
-!!! warning
-    Unlike AWS and GCP, Azure requires its *virtual* networks to have non-overlapping IPv4 CIDRs (yeah, go figure). Instead of each cluster just using `10.0.0.0/16` for instances, each Azure cluster's `host_cidr` must be non-overlapping (e.g. 10.0.0.0/20 for the 1st cluster, 10.0.16.0/20 for the 2nd cluster, etc).
-
 !!! warning
     Do not choose a `controller_type` smaller than `Standard_B2s`. Smaller instances are not sufficient for running a controller.
 

docs/flatcar-linux/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.30.3 cluster on bare-metal with Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.31.0 cluster on bare-metal with Flatcar Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -154,7 +154,7 @@ Define a Kubernetes cluster using the module `bare-metal/flatcar-linux/kubernete
 
 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.31.0"
 
   # bare-metal
   cluster_name            = "mercury"
@@ -194,7 +194,7 @@ Workers with similar features can be defined inline using the `workers` field as
 
 ```tf
 module "mercury-node1" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes/worker?ref=v1.31.0"
 
   # bare-metal
   cluster_name = "mercury"
@@ -323,9 +323,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.30.3
+node1.example.com   Ready   <none>  10m  v1.31.0
-node2.example.com   Ready   <none>  10m  v1.30.3
+node2.example.com   Ready   <none>  10m  v1.31.0
-node3.example.com   Ready   <none>  10m  v1.30.3
+node3.example.com   Ready   <none>  10m  v1.31.0
 ```
 
 List the pods.
@@ -333,9 +333,10 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
-kube-system   calico-node-6qp7f                          2/2       Running   1          11m
+kube-system   cilium-6qp7f                               1/1       Running   1          11m
-kube-system   calico-node-gnjrm                          2/2       Running   0          11m
+kube-system   cilium-gnjrm                               1/1       Running   0          11m
-kube-system   calico-node-llbgt                          2/2       Running   0          11m
+kube-system   cilium-llbgt                               1/1       Running   0          11m
+kube-system   cilium-operator-68d778b448-g744f           1/1       Running   0          11m
 kube-system   coredns-1187388186-dj3pd                   1/1       Running   0          11m
 kube-system   coredns-1187388186-mx9rt                   1/1       Running   0          11m
 kube-system   kube-apiserver-node1.example.com           1/1       Running   0          11m

docs/flatcar-linux/digitalocean.md

@@ -1,6 +1,6 @@
 # DigitalOcean
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on DigitalOcean with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on DigitalOcean with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -81,19 +81,19 @@ Define a Kubernetes cluster using the module `digital-ocean/flatcar-linux/kubern
 
 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=v1.31.0"
 
   # Digital Ocean
   cluster_name = "nemo"
   region       = "nyc3"
   dns_zone     = "digital-ocean.example.com"
 
-  # configuration
+  # instances
-  os_image         = data.digitalocean_image.flatcar-stable-2303-4-0.id
+  os_image     = data.digitalocean_image.flatcar-stable-2303-4-0.id
-  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
-
-  # optional
   worker_count = 2
+
+  # configuration
+  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
 }
 ```
 
@@ -155,9 +155,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.30.3
+10.132.110.130     Ready   <none>  10m  v1.31.0
-10.132.115.81      Ready   <none>  10m  v1.30.3
+10.132.115.81      Ready   <none>  10m  v1.31.0
-10.132.124.107     Ready   <none>  10m  v1.30.3
+10.132.124.107     Ready   <none>  10m  v1.31.0
 ```
 
 List the pods.
@@ -166,9 +166,9 @@ List the pods.
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
 kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
-kube-system   calico-node-1m5bf                          2/2       Running   0          11m
+kube-system   cilium-1m5bf                               1/1       Running   0          11m
-kube-system   calico-node-7jmr1                          2/2       Running   0          11m
+kube-system   cilium-7jmr1                               1/1       Running   0          11m
-kube-system   calico-node-bknc8                          2/2       Running   0          11m
+kube-system   cilium-bknc8                               1/1       Running   0          11m
 kube-system   kube-apiserver-ip-10.132.115.81            1/1       Running   0          11m
 kube-system   kube-controller-manager-ip-10.132.115.81   1/1       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m

docs/flatcar-linux/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.30.3 cluster on Google Compute Engine with Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.31.0 cluster on Google Compute Engine with Flatcar Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -73,7 +73,7 @@ Define a Kubernetes cluster using the module `google-cloud/flatcar-linux/kuberne
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/flatcar-linux/kubernetes?ref=v1.31.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -81,11 +81,11 @@ module "yavin" {
   dns_zone      = "example.com"
   dns_zone_name = "example-zone"
 
+  # instances
+  worker_count = 2
+
   # configuration
   ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-
-  # optional
-  worker_count = 2
 }
 ```
 
@@ -147,9 +147,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.31.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.31.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.31.0
 ```
 
 List the pods.
@@ -157,9 +157,9 @@ List the pods.
 ```
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
-kube-system   calico-node-1cs8z                         2/2    Running   0         6m
+kube-system   cilium-1cs8z                              1/1    Running   0         6m
-kube-system   calico-node-d1l5b                         2/2    Running   0         6m
+kube-system   cilium-d1l5b                              1/1    Running   0         6m
-kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   cilium-sp9ps                              1/1    Running   0         6m
 kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-controller-0               1/1    Running   0         6m
@@ -211,12 +211,13 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
-| controller_count | Number of controllers (i.e. masters) | 1 | 3 |
-| worker_count | Number of workers | 1 | 3 |
-| controller_type | Machine type for controllers | "n1-standard-1" | See below |
-| worker_type | Machine type for workers | "n1-standard-1" | See below |
 | os_image | Flatcar Linux image for compute instances | "flatcar-stable" | flatcar-stable, flatcar-beta, flatcar-alpha |
-| disk_size | Size of the disk in GB | 30 | 100 |
+| controller_count | Number of controllers (i.e. masters) | 1 | 3 |
+| controller_type | Machine type for controllers | "n1-standard-1" | See below |
+| controller_disk_type | Controller disk size in GB | 30 | 20 |
+| worker_count | Number of workers | 1 | 3 |
+| worker_type | Machine type for workers | "n1-standard-1" | See below |
+| worker_disk_size | Worker disk size in GB | 30 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](fedora-coreos/google-cloud/#preemption) workers, and [snippets](advanced/customization/#hosts) customization
@@ -70,7 +70,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
 
   # Google Cloud
   cluster_name  = "yavin"
@@ -108,9 +108,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.30.3
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.31.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.31.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.30.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.31.0
 ```
 
 List the pods.

docs/topics/maintenance.md

@@ -13,12 +13,12 @@ Typhoon provides tagged releases to allow clusters to be versioned using ordinar
 
 ```
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.31.0"
   ...
 }
 
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.30.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/flatcar-linux/kubernetes?ref=v1.31.0"
   ...
 }
 ```
@@ -192,7 +192,7 @@ Applying edits to most worker fields will start an instance refresh:
 However, changing `os_stream`/`os_channel` or new AMIs becoming available will NOT change the launch configuration or trigger an Instance Refresh. This allows Fedora CoreOS or Flatcar Linux to auto-update themselves via reboots and avoids unexpected terraform diffs for new AMIs.
 
 !!! note
-    Before Typhoon v1.30.3, worker nodes only used new launch configurations when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
+    Before Typhoon v1.31.0, worker nodes only used new launch configurations when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
 
 ### Google Cloud
 
@@ -233,7 +233,7 @@ Applying edits to most worker fields will start an instance refresh:
 However, changing `os_stream`/`os_channel` or new compute images becoming available will NOT change the launch template or update instances. This allows Fedora CoreOS or Flatcar Linux to auto-update themselves via reboots and avoids unexpected terraform diffs for new AMIs.
 
 !!! note
-    Before Typhoon v1.30.3, worker nodes only used new launch templates when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
+    Before Typhoon v1.31.0, worker nodes only used new launch templates when replaced manually (or due to failure). If you must change node configuration manually, it's still possible. Create a new [worker pool](../advanced/worker-pools.md), then scale down the old worker pool as desired.
 
 ## Upgrade poseidon/ct
 

google-cloud/fedora-coreos/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.30.3 (upstream)
+* Kubernetes v1.31.0 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/fedora-coreos/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization

google-cloud/fedora-coreos/kubernetes/bootstrap.tf

@@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1609060f4f138f3b3aef74a9e5494e0fe831c423"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ddecb1cef65c9715ed66b6c335634bc51f59613"
 
   cluster_name          = var.cluster_name
   api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
@@ -9,9 +9,6 @@ module "bootstrap" {
   network_mtu           = 1440
   pod_cidr              = var.pod_cidr
   service_cidr          = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
-  enable_reporting      = var.enable_reporting
-  enable_aggregation    = var.enable_aggregation
   daemonset_tolerations = var.daemonset_tolerations
   components            = var.components
 

google-cloud/fedora-coreos/kubernetes/butane/controller.yaml

@@ -54,7 +54,7 @@ systemd:
         Description=Kubelet (System Container)
         Wants=rpc-statd.service
         [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.30.3
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.31.0
         ExecStartPre=/bin/mkdir -p /etc/cni/net.d
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -111,7 +111,7 @@ systemd:
             --volume /opt/bootstrap/assets:/assets:ro,Z \
             --volume /opt/bootstrap/apply:/apply:ro,Z \
             --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.30.3
+            quay.io/poseidon/kubelet:v1.31.0
         ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
         ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@@ -143,7 +143,7 @@ storage:
           cgroupDriver: systemd
           clusterDNS:
             - ${cluster_dns_service_ip}
-          clusterDomain: ${cluster_domain_suffix}
+          clusterDomain: cluster.local
           healthzPort: 0
           rotateCertificates: true
           shutdownGracePeriod: 45s

google-cloud/fedora-coreos/kubernetes/controllers.tf

@@ -81,7 +81,6 @@ data "ct_config" "controllers" {
     kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
     ssh_authorized_key     = var.ssh_authorized_key
     cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
-    cluster_domain_suffix  = var.cluster_domain_suffix
   })
   strict   = true
   snippets = var.controller_snippets

google-cloud/fedora-coreos/kubernetes/variables.tf

@@ -115,32 +115,13 @@ EOD
   default     = "10.3.0.0/16"
 }
 
-
-variable "enable_reporting" {
-  type        = bool
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = false
-}
-
-variable "enable_aggregation" {
-  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer"
-  default     = true
-}
-
 variable "worker_node_labels" {
   type        = list(string)
   description = "List of initial worker node labels"
   default     = []
 }
 
-# unofficial, undocumented, unsupported
+# advanced
-
-variable "cluster_domain_suffix" {
-  type        = string
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  default     = "cluster.local"
-}
 
 variable "daemonset_tolerations" {
   type        = list(string)

google-cloud/fedora-coreos/kubernetes/workers.tf

@@ -13,11 +13,10 @@ module "workers" {
   preemptible  = var.worker_preemptible
 
   # configuration
-  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  kubeconfig         = module.bootstrap.kubeconfig-kubelet
-  ssh_authorized_key    = var.ssh_authorized_key
+  ssh_authorized_key = var.ssh_authorized_key
-  service_cidr          = var.service_cidr
+  service_cidr       = var.service_cidr
-  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets           = var.worker_snippets
-  snippets              = var.worker_snippets
+  node_labels        = var.worker_node_labels
-  node_labels           = var.worker_node_labels
 }