Update mkdocs-material docs theme version

* https://github.com/grafana/grafana/releases/tag/v5.3.2

CHANGES.md

@@ -4,6 +4,74 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.12.2
+
+* Kubernetes [v1.12.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122)
+* Update CoreDNS from 1.2.2 to [1.2.4](https://github.com/coredns/coredns/releases/tag/v1.2.4)
+* Update Calico from v3.2.3 to [v3.3.0](https://docs.projectcalico.org/v3.3/releases/)
+* Disable Kubelet read-only port ([#324](https://github.com/poseidon/typhoon/pull/324))
+* Fix CoreDNS AntiAffinity spec to prefer spreading replicas
+* Ignore controller node user-data changes ([#335](https://github.com/poseidon/typhoon/pull/335))
+  * Once all managed clusters use v1.12.2, it is possible to update `terraform-provider-ct`
+
+#### AWS
+
+* Add `disk_iops` variable for EBS volume IOPS ([#314](https://github.com/poseidon/typhoon/pull/314))
+
+#### Azure
+
+* Use new `azurerm_network_interface_backend_address_pool_association` ([#332](https://github.com/poseidon/typhoon/pull/332))
+  * Require `terraform-provider-azurerm` v1.17+ (action required)
+* Add `primary` field to `ip_configuration` needed by v1.17+ ([#331](https://github.com/poseidon/typhoon/pull/331))
+
+#### DigitalOcean
+
+* Add AAAA DNS records resolving to worker nodes ([#333](https://github.com/poseidon/typhoon/pull/333))
+  * Hosting IPv6 apps requires editing nginx-ingress with `hostNetwork: true`
+
+#### Google Cloud
+
+* Add an IPv6 address and IPv6 forwarding rules for load balancing IPv6 Ingress ([#334](https://github.com/poseidon/typhoon/pull/334))
+  * Add `ingress_static_ipv6` output variable for use in AAAA DNS records
+  * Allow serving IPv6 applications via Kubernetes Ingress
+
+#### Addons
+
+* Configure Heapster to scrape Kubelets with bearer token auth ([#323](https://github.com/poseidon/typhoon/pull/323))
+* Update Grafana from v5.3.1 to v5.3.2
+
+## v1.12.1
+
+* Kubernetes [v1.12.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1121)
+* Update etcd from v3.3.9 to [v3.3.10](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10)
+* Update CoreDNS from 1.1.3 to [1.2.2](https://github.com/coredns/coredns/releases/tag/v1.2.2)
+* Update Calico from v3.2.1 to [v3.2.3](https://docs.projectcalico.org/v3.2/releases/)
+* Raise scheduler and controller-manager replicas to the larger of 2 or the number of controller nodes ([#312](https://github.com/poseidon/typhoon/pull/312))
+  * Single-controller clusters continue to run 2 replicas as before
+* Raise default CoreDNS replicas to the larger of 2 or the number of controller nodes ([#313](https://github.com/poseidon/typhoon/pull/313))
+  * Add AntiAffinity preferred rule to favor spreading CoreDNS pods
+* Annotate control plane and addon containers to use the Docker runtime seccomp profile ([#319](https://github.com/poseidon/typhoon/pull/319))
+  * Override Kubernetes default behavior that starts containers with `seccomp=unconfined`
+
+#### Azure
+
+* Remove `admin_password` field (disabled) since it is now optional
+  * Require `terraform-provider-azurerm` v1.16+ (action required)
+
+#### Bare-Metal
+
+* Add support for `cached_install` mode with Flatcar Linux ([#315](https://github.com/poseidon/typhoon/pull/315))
+
+#### DigitalOcean
+
+* Require `terraform-provider-digitalocean` v1.0+ (action required)
+
+#### Addons
+
+* Update nginx-ingress from v0.19.0 to v0.20.0
+* Update Prometheus from v2.3.2 to v2.4.3
+* Update Grafana from v5.2.4 to v5.3.1
+
 ## v1.11.3
 
 * Kubernetes [v1.11.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113)

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
@@ -39,7 +39,7 @@ The AWS and bare-metal `container-linux` modules allow picking Red Hat Container
 
 * [Docs](https://typhoon.psdn.io)
 * Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
-* Tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [Digital Ocean](cl/digital-ocean.md), and [Google-Cloud](cl/google-cloud.md)
+* Tutorials for [AWS](docs/cl/aws.md), [Azure](docs/cl/azure.md), [Bare-Metal](docs/cl/bare-metal.md), [Digital Ocean](docs/cl/digital-ocean.md), and [Google-Cloud](docs/cl/google-cloud.md)
 
 ## Usage
 
@@ -47,7 +47,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
   
   providers = {
     google   = "google.default"
@@ -88,9 +88,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.11.3
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.11.3
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.11.3
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
 ```
 
 List the pods.

addons/cluo/update-agent.yaml

@@ -15,6 +15,8 @@ spec:
     metadata:
       labels:
         app: container-linux-update-agent
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
       - name: update-agent

addons/cluo/update-operator.yaml

@@ -12,6 +12,8 @@ spec:
     metadata:
       labels:
         app: container-linux-update-operator
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
       - name: update-operator

addons/grafana/deployment.yaml

@@ -18,10 +18,12 @@ spec:
       labels:
         name: grafana
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:5.2.4
+          image: grafana/grafana:5.3.2
           env:
             - name: GF_SERVER_HTTP_PORT
               value: "8080"

addons/heapster/cluster-role-binding.yaml

@@ -5,7 +5,7 @@ metadata:
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
-  name: system:heapster
+  name: heapster
 subjects:
 - kind: ServiceAccount
   name: heapster

addons/heapster/cluster-role.yaml

@@ -0,0 +1,30 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: heapster
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - events
+  - namespaces
+  - nodes
+  - pods
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - deployments
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - nodes/stats
+  verbs:
+  - get

addons/heapster/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           image: k8s.gcr.io/heapster-amd64:v1.5.4
           command:
             - /heapster
-            - --source=kubernetes.summary_api:''
+            - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
           livenessProbe:
             httpGet:
               path: /healthz

addons/nginx-ingress/aws/default-backend/deployment.yaml

@@ -14,6 +14,8 @@ spec:
       labels:
         name: default-backend
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: default-backend

addons/nginx-ingress/aws/deployment.yaml

@@ -17,12 +17,14 @@ spec:
       labels:
         name: nginx-ingress-controller
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/azure/default-backend/deployment.yaml

@@ -14,6 +14,8 @@ spec:
       labels:
         name: default-backend
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: default-backend

addons/nginx-ingress/azure/deployment.yaml

@@ -17,12 +17,14 @@ spec:
       labels:
         name: nginx-ingress-controller
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/bare-metal/default-backend/deployment.yaml

@@ -14,6 +14,8 @@ spec:
       labels:
         name: default-backend
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: default-backend

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -17,10 +17,12 @@ spec:
       labels:
         name: ingress-controller-public
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -17,12 +17,14 @@ spec:
       labels:
         name: nginx-ingress-controller
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/digital-ocean/default-backend/deployment.yaml

@@ -14,6 +14,8 @@ spec:
       labels:
         name: default-backend
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: default-backend

addons/nginx-ingress/google-cloud/default-backend/deployment.yaml

@@ -14,6 +14,8 @@ spec:
       labels:
         name: default-backend
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       containers:
         - name: default-backend

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -17,12 +17,14 @@ spec:
       labels:
         name: nginx-ingress-controller
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/prometheus/deployment.yaml

@@ -14,11 +14,13 @@ spec:
       labels:
         name: prometheus
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.3.2
+          image: quay.io/prometheus/prometheus:v2.4.3
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -18,6 +18,8 @@ spec:
       labels:
         name: kube-state-metrics
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       serviceAccountName: kube-state-metrics
       containers:

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -17,6 +17,8 @@ spec:
       labels:
         name: node-exporter
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       serviceAccountName: node-exporter
       securityContext:

aws/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -88,6 +88,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -122,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

aws/container-linux/kubernetes/controllers.tf

@@ -24,12 +24,13 @@ resource "aws_instance" "controllers" {
   instance_type = "${var.controller_type}"
 
   ami       = "${local.ami_id}"
-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
 
   # storage
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -38,12 +39,23 @@ resource "aws_instance" "controllers" {
   vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
 
   lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
   }
 }
 
-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
   count = "${var.controller_count}"
 
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@@ -73,10 +85,3 @@ data "template_file" "etcds" {
     dns_zone     = "${var.dns_zone}"
   }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
-}

aws/container-linux/kubernetes/security.tf

@@ -104,27 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -300,27 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
   security_group_id = "${aws_security_group.worker.id}"
 

aws/container-linux/kubernetes/variables.tf

@@ -59,6 +59,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
   type        = "string"
   default     = ""

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -60,6 +60,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -92,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -110,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.11.3 \
+            docker://k8s.gcr.io/hyperkube:v1.12.2 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/variables.tf

@@ -52,6 +52,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
   type        = "string"
   default     = ""

aws/container-linux/kubernetes/workers/workers.tf

@@ -46,12 +46,13 @@ resource "aws_launch_configuration" "worker" {
   spot_price        = "${var.spot_price}"
   enable_monitoring = false
 
-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"
 
   # storage
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -64,8 +65,15 @@ resource "aws_launch_configuration" "worker" {
   }
 }
 
-# Worker Container Linux Config
-data "template_file" "worker_config" {
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.clc_snippets}"]
+}
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
@@ -75,9 +83,3 @@ data "template_file" "worker_config" {
     cluster_domain_suffix = "${var.cluster_domain_suffix}"
   }
 }
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
-  pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
-}

aws/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -65,6 +65,7 @@ write_files:
         --node-labels=node-role.kubernetes.io/master \
         --node-labels=node-role.kubernetes.io/controller="true" \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/kubernetes/kubeconfig
@@ -92,8 +93,8 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, cloud-metadata.service]

aws/fedora-atomic/kubernetes/controllers.tf

@@ -30,6 +30,7 @@ resource "aws_instance" "controllers" {
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -38,7 +39,10 @@ resource "aws_instance" "controllers" {
   vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
 
   lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
   }
 }
 

aws/fedora-atomic/kubernetes/security.tf

@@ -104,27 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -300,27 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
   security_group_id = "${aws_security_group.worker.id}"
 

aws/fedora-atomic/kubernetes/variables.tf

@@ -53,6 +53,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
   type        = "string"
   default     = ""

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -43,6 +43,7 @@ write_files:
         --network-plugin=cni \
         --node-labels=node-role.kubernetes.io/node \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/kubernetes/kubeconfig
     permissions: '0644'
@@ -69,7 +70,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -46,6 +46,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
   type        = "string"
   default     = ""

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -52,6 +52,7 @@ resource "aws_launch_configuration" "worker" {
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network

azure/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

azure/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -88,6 +88,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -122,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

azure/container-linux/kubernetes/controllers.tf

@@ -85,6 +85,7 @@ resource "azurerm_virtual_machine" "controllers" {
   lifecycle {
     ignore_changes = [
       "storage_os_disk",
+      "os_profile",
     ]
   }
 }
@@ -105,12 +106,16 @@ resource "azurerm_network_interface" "controllers" {
 
     # public IPv4
     public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
-
-    # backend address pool to which the NIC should be added
-    load_balancer_backend_address_pools_ids = ["${azurerm_lb_backend_address_pool.controller.id}"]
   }
 }
 
+# Add controller NICs to the controller backend address pool
+resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+  network_interface_id    = "${azurerm_network_interface.controllers.id}"
+  ip_configuration_name   = "ip0"
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+}
+
 # Controller public IPv4 addresses
 resource "azurerm_public_ip" "controllers" {
   count               = "${var.controller_count}"

azure/container-linux/kubernetes/require.tf

@@ -0,0 +1,25 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = ">= 0.11.0"
+}
+
+provider "azurerm" {
+  version = "~> 1.17"
+}
+
+provider "local" {
+  version = "~> 1.0"
+}
+
+provider "null" {
+  version = "~> 1.0"
+}
+
+provider "template" {
+  version = "~> 1.0"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+}

azure/container-linux/kubernetes/security.tf

@@ -117,22 +117,6 @@ resource "azurerm_network_security_rule" "controller-kubelet" {
   destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "azurerm_network_security_rule" "controller-kubelet-read" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
-
-  name                        = "allow-kubelet-read"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
-  priority                    = "2035"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10255"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
-}
-
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
 
@@ -269,22 +253,6 @@ resource "azurerm_network_security_rule" "worker-kubelet" {
   destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "azurerm_network_security_rule" "worker-kubelet-read" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
-
-  name                        = "allow-kubelet-read"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
-  priority                    = "2030"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10255"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
-}
-
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
 

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -60,6 +60,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -92,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -110,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.11.3 \
+            docker://k8s.gcr.io/hyperkube:v1.12.2 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')

azure/container-linux/kubernetes/workers/ingress.tf

@@ -1 +0,0 @@
-

azure/container-linux/kubernetes/workers/workers.tf

@@ -37,10 +37,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
   os_profile {
     computer_name_prefix = "${var.name}-worker-"
     admin_username       = "core"
-
-    # Required by Azure, but password auth is disabled below
-    admin_password = ""
-    custom_data    = "${element(data.ct_config.worker-ignitions.*.rendered, count.index)}"
+    custom_data          = "${data.ct_config.worker-ignition.rendered}"
   }
 
   # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@@ -61,6 +58,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
 
     ip_configuration {
       name      = "ip0"
+      primary   = true
       subnet_id = "${var.subnet_id}"
 
       # backend address pool to which the NIC should be added
@@ -96,14 +94,14 @@ resource "azurerm_autoscale_setting" "workers" {
 }
 
 # Worker Ignition configs
-data "ct_config" "worker-ignitions" {
-  content      = "${data.template_file.worker-configs.rendered}"
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
   pretty_print = false
   snippets     = ["${var.clc_snippets}"]
 }
 
 # Worker Container Linux configs
-data "template_file" "worker-configs" {
+data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {

bare-metal/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name                    = "${var.cluster_name}"
   api_servers                     = ["${var.k8s_domain_name}"]

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@@ -97,6 +97,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -123,7 +124,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -69,6 +69,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -84,7 +85,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/groups.tf

@@ -3,7 +3,7 @@ resource "matchbox_group" "install" {
 
   name = "${format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
 
-  profile = "${local.flavor == "flatcar" ? element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
+  profile = "${local.flavor == "flatcar" ? var.cached_install == "true" ? element(matchbox_profile.cached-flatcar-linux-install.*.name, count.index) : element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
 
   selector {
     mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"

bare-metal/container-linux/kubernetes/profiles.tf

@@ -49,7 +49,7 @@ data "template_file" "container-linux-install-configs" {
 }
 
 // Container Linux Install profile (from matchbox /assets cache)
-// Note: Admin must have downloaded os_version into matchbox assets.
+// Note: Admin must have downloaded os_version into matchbox assets/coreos.
 resource "matchbox_profile" "cached-container-linux-install" {
   count = "${length(var.controller_names) + length(var.worker_names)}"
   name  = "${format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
@@ -87,7 +87,7 @@ data "template_file" "cached-container-linux-install-configs" {
     ssh_authorized_key  = "${var.ssh_authorized_key}"
 
     # profile uses -b baseurl to install from matchbox cache
-    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/coreos"
+    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/${local.flavor}"
   }
 }
 
@@ -114,6 +114,30 @@ resource "matchbox_profile" "flatcar-install" {
   container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
 }
 
+// Flatcar Linux Install profile (from matchbox /assets cache)
+// Note: Admin must have downloaded os_version into matchbox assets/flatcar.
+resource "matchbox_profile" "cached-flatcar-linux-install" {
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+  name  = "${format("%s-cached-flatcar-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+
+  kernel = "/assets/flatcar/${var.os_version}/flatcar_production_pxe.vmlinuz"
+
+  initrd = [
+    "/assets/flatcar/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
+  ]
+
+  args = [
+    "initrd=flatcar_production_pxe_image.cpio.gz",
+    "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
+    "flatcar.first_boot=yes",
+    "console=tty0",
+    "console=ttyS0",
+    "${var.kernel_args}",
+  ]
+
+  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
+}
+
 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
   count        = "${length(var.controller_names)}"

bare-metal/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${var.k8s_domain_name}"]

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -51,6 +51,7 @@ write_files:
         --node-labels=node-role.kubernetes.io/master \
         --node-labels=node-role.kubernetes.io/controller="true" \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
@@ -83,8 +84,8 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -29,6 +29,7 @@ write_files:
         --network-plugin=cni \
         --node-labels=node-role.kubernetes.io/node \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
     content: |
@@ -59,7 +60,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -100,6 +100,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -128,7 +129,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -72,6 +72,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -98,7 +99,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -116,7 +117,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.11.3 \
+            docker://k8s.gcr.io/hyperkube:v1.12.2 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/container-linux/kubernetes/controllers.tf

@@ -44,12 +44,18 @@ resource "digitalocean_droplet" "controllers" {
   ipv6               = true
   private_networking = true
 
-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
   ssh_keys  = ["${var.ssh_fingerprints}"]
 
   tags = [
     "${digitalocean_tag.controllers.id}",
   ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }
 
 # Tag to label controllers
@@ -57,8 +63,16 @@ resource "digitalocean_tag" "controllers" {
   name = "${var.cluster_name}-controller"
 }
 
-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
   count = "${var.controller_count}"
 
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@@ -85,11 +99,3 @@ data "template_file" "etcds" {
     dns_zone     = "${var.dns_zone}"
   }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-
-  snippets = ["${var.controller_clc_snippets}"]
-}

digital-ocean/container-linux/kubernetes/outputs.tf

@@ -3,7 +3,8 @@ output "controllers_dns" {
 }
 
 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }
 
 output "controllers_ipv4" {

digital-ocean/container-linux/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "digitalocean" {
-  version = "~> 0.1.2"
+  version = "~> 1.0"
 }
 
 provider "local" {

digital-ocean/container-linux/kubernetes/workers.tf

@@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
   count = "${var.worker_count}"
 
   # DNS zone where record should be created
@@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
   value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }
 
+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
   count = "${var.worker_count}"
@@ -25,12 +37,16 @@ resource "digitalocean_droplet" "workers" {
   ipv6               = true
   private_networking = true
 
-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"
   ssh_keys  = ["${var.ssh_fingerprints}"]
 
   tags = [
     "${digitalocean_tag.workers.id}",
   ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }
 
 # Tag to label workers
@@ -38,8 +54,15 @@ resource "digitalocean_tag" "workers" {
   name = "${var.cluster_name}-worker"
 }
 
-# Worker Container Linux Config
-data "template_file" "worker_config" {
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.worker_clc_snippets}"]
+}
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
@@ -47,9 +70,3 @@ data "template_file" "worker_config" {
     cluster_domain_suffix = "${var.cluster_domain_suffix}"
   }
 }
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
-  pretty_print = false
-  snippets     = ["${var.worker_clc_snippets}"]
-}

digital-ocean/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

digital-ocean/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -65,6 +65,7 @@ write_files:
         --node-labels=node-role.kubernetes.io/master \
         --node-labels=node-role.kubernetes.io/controller="true" \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
@@ -89,8 +90,8 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, cloud-metadata.service]

digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -43,6 +43,7 @@ write_files:
         --network-plugin=cni \
         --node-labels=node-role.kubernetes.io/node \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
     content: |
@@ -66,7 +67,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/controllers.tf

@@ -50,6 +50,12 @@ resource "digitalocean_droplet" "controllers" {
   tags = [
     "${digitalocean_tag.controllers.id}",
   ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }
 
 # Tag to label controllers

digital-ocean/fedora-atomic/kubernetes/outputs.tf

@@ -3,7 +3,8 @@ output "controllers_dns" {
 }
 
 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }
 
 output "controllers_ipv4" {

digital-ocean/fedora-atomic/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "digitalocean" {
-  version = "~> 0.1.2"
+  version = "~> 1.0"
 }
 
 provider "local" {

digital-ocean/fedora-atomic/kubernetes/workers.tf

@@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
   count = "${var.worker_count}"
 
   # DNS zone where record should be created
@@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
   value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }
 
+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
   count = "${var.worker_count}"
@@ -31,6 +43,10 @@ resource "digitalocean_droplet" "workers" {
   tags = [
     "${digitalocean_tag.workers.id}",
   ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }
 
 # Tag to label workers

docs/addons/ingress.md

@@ -4,7 +4,7 @@ Nginx Ingress controller pods accept and demultiplex HTTP, HTTPS, TCP, or UDP tr
 
 ## AWS
 
-On AWS, a network load balancer (NLB) distributes traffic across a target group of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.
+On AWS, a network load balancer (NLB) distributes TCP traffic across two target groups (port 80 and 443) of worker nodes running an Ingress controller deployment. Security groups rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -37,7 +37,7 @@ resource "google_dns_record_set" "some-application" {
 
 ## Azure
 
-On Azure, a load balancer distributes traffic across a backend pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
+On Azure, a load balancer distributes traffic across a backend address pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -101,7 +101,7 @@ resource "google_dns_record_set" "some-application" {
 
 ## Digital Ocean
 
-On Digital Ocean, a DNS A record (e.g. `nemo-workers.example.com`) resolves to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
+On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
 
 Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -124,11 +124,14 @@ resource "google_dns_record_set" "some-application" {
 }
 ```
 
+!!! note
+    Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
+
 [^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
 
 ## Google Cloud
 
-On Google Cloud, a TCP Proxy load balancer distributes traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.
+On Google Cloud, a TCP Proxy load balancer distributes IPv4 and IPv6 TCP traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -136,7 +139,7 @@ Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, de
 kubectl apply -R -f addons/nginx-ingress/google-cloud
 ```
 
-For each application, add a DNS record resolving to the load balancer's IPv4 address.
+For each application, add DNS A records resolving to the load balancer's IPv4 address and DNS AAAA records resolving to the load balancer's IPv6 address.
 
 ```
 app1.example.com -> 11.22.33.44
@@ -144,10 +147,10 @@ app2.example.com -> 11.22.33.44
 app3.example.com -> 11.22.33.44
 ```
 
-Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's output `ingress_static_ipv4`. For example, you might use Terraform to manage a Google Cloud DNS record:
+Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's outputs `ingress_static_ipv4` and `ingress_static_ipv6`. For example, you might use Terraform to manage a Google Cloud DNS record:
 
 ```tf
-resource "google_dns_record_set" "some-application" {
+resource "google_dns_record_set" "app-record-a" {
   # DNS zone name
   managed_zone = "example-zone"
 
@@ -157,4 +160,15 @@ resource "google_dns_record_set" "some-application" {
   ttl     = 300
   rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv4}"]
 }
+
+resource "google_dns_record_set" "app-record-aaaa" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "AAAA"
+  ttl     = 300
+  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv6}"]
+}
 ```

docs/addons/prometheus.md

@@ -47,4 +47,4 @@ Visit [127.0.0.1:9090](http://127.0.0.1:9090) to query [expressions](http://127.
 <br/>
 ![Prometheus Alerts](../img/prometheus-alerts.png)
 
-Use [Grafana](/addons/grafana.md) to view or build dashboards that use Prometheus as the datasource.
+Use [Grafana](/addons/grafana/) to view or build dashboards that use Prometheus as the datasource.

docs/advanced/worker-pools.md

@@ -16,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo
 
 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.12.2"
   
   providers = {
     aws = "aws.default"
@@ -82,7 +82,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define
 
 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.12.2"
   
   providers = {
     azurerm = "azurerm.default"
@@ -152,7 +152,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu
 
 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.12.2"
 
   providers = {
     google = "google.default"
@@ -187,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.11.3
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.11.3
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.11.3
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.11.3
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.11.3
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.12.2
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.12.2
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.12.2
 ```
 
 ### Variables

docs/architecture/aws.md

@@ -0,0 +1,13 @@
+# AWS
+
+## IPv6
+
+Status of IPv6 on Typhoon AWS clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | No        |
+
+* AWS Network Load Balancers do not support `dualstack`.

docs/architecture/azure.md

@@ -0,0 +1,13 @@
+# Azure
+
+## IPv6
+
+Status of IPv6 on Typhoon Azure clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | No        |
+| Node Outbound IPv6      | No        |
+| Kubernetes Ingress IPv6 | No        |
+
+* Azure does not allow reserving a static IPv6 address 

docs/architecture/bare-metal.md

@@ -0,0 +1,13 @@
+# Bare-Metal
+
+## IPv6
+
+Status of IPv6 on Typhoon bare-metal clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Possible  |
+
+IPv6 support depends upon the bare-metal network environment.

docs/architecture/digitalocean.md

@@ -0,0 +1,11 @@
+# AWS
+
+## IPv6
+
+Status of IPv6 on Typhoon DigitalOcean clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Possible  |

docs/architecture/google-cloud.md

@@ -0,0 +1,11 @@
+# Google Cloud
+
+## IPv6
+
+Status of IPv6 on Typhoon Google Cloud clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | No        |
+| Node Outbound IPv6      | No        |
+| Kubernetes Ingress IPv6 | Yes       |

docs/atomic/aws.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on AWS with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on AWS with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -24,7 +24,7 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `aws/fedora-atomic/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.12.2"
 
   providers = {
     aws = "aws.default"
@@ -156,9 +156,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.11.3
-ip-10-0-19-112   Ready     34m       v1.11.3
-ip-10-0-4-22     Ready     34m       v1.11.3
+ip-10-0-12-221   Ready     34m       v1.12.2
+ip-10-0-19-112   Ready     34m       v1.12.2
+ip-10-0-4-22     Ready     34m       v1.12.2
 ```
 
 List the pods.
@@ -184,7 +184,7 @@ kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 ## Variables
 
@@ -227,6 +227,7 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | worker_type | EC2 instance type for workers | "t2.small" | See below |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| disk_iops | IOPS of the EBS volume | "0" (i.e. auto) | "400" |
 | worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |

docs/atomic/bare-metal.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.11.3 cluster on bare-metal with Fedora Atomic.
+In this tutorial, we'll network boot and provision a Kubernetes v1.12.2 cluster on bare-metal with Fedora Atomic.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora Atomic via kickstart, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via cloud-init.
 
@@ -95,7 +95,7 @@ For networks already supporting iPXE clients, you can add a `default.ipxe` confi
 chain http://matchbox.foo:8080/boot.ipxe
 ```
 
-For networks with Ubiquiti Routers, you can [configure the router](/topics/hardware.md#ubiquiti) itself to chainload machines to iPXE and Matchbox.
+For networks with Ubiquiti Routers, you can [configure the router](/topics/hardware/#ubiquiti) itself to chainload machines to iPXE and Matchbox.
 
 For a small lab, you may wish to checkout the [quay.io/coreos/dnsmasq](https://quay.io/repository/coreos/dnsmasq) container image and [copy-paste examples](https://github.com/coreos/matchbox/blob/master/Documentation/network-setup.md#coreosdnsmasq).
 
@@ -190,7 +190,7 @@ providers {
 }
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -235,7 +235,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-atomic/kubernete
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.12.2"
   
   providers = {
     local = "local.default"
@@ -361,9 +361,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.11.3
-node2.example.com   Ready     11m       v1.11.3
-node3.example.com   Ready     11m       v1.11.3
+node1.example.com   Ready     11m       v1.12.2
+node2.example.com   Ready     11m       v1.12.2
+node3.example.com   Ready     11m       v1.12.2
 ```
 
 List the pods.
@@ -389,7 +389,7 @@ kube-system   pod-checkpointer-wf65d-node1.example.com   1/1       Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 ## Variables
 

docs/atomic/digital-ocean.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on DigitalOcean with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on DigitalOcean with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -24,7 +24,7 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -45,7 +45,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.
 
 ```tf
 provider "digitalocean" {
-  version = "0.1.3"
+  version = "1.0.0"
   token = "${chomp(file("~/.config/digital-ocean/token"))}"
   alias = "default"
 }
@@ -77,7 +77,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-atomic/kubern
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.12.2"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -152,9 +152,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.11.3
-10.132.115.81    Ready     10m       v1.11.3
-10.132.124.107   Ready     10m       v1.11.3
+10.132.110.130   Ready     10m       v1.12.2
+10.132.115.81    Ready     10m       v1.12.2
+10.132.124.107   Ready     10m       v1.12.2
 ```
 
 List the pods.
@@ -179,7 +179,7 @@ kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 ## Variables
 

docs/atomic/google-cloud.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Fedora does not publish official images for Google Cloud so you must prepare them yourself. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on Google Compute Engine with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on Google Compute Engine with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -25,7 +25,7 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -121,7 +121,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-atomic/kuberne
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.12.2"
   
   providers = {
     google   = "google.default"
@@ -197,9 +197,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.11.3
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.11.3
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.11.3
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
 ```
 
 List the pods.
@@ -224,7 +224,7 @@ kube-system   pod-checkpointer-l6lrt                    1/1    Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 ## Variables
 

docs/cl/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on AWS with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -37,7 +37,7 @@ providers {
 }
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -96,7 +96,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.12.2"
 
   providers = {
     aws = "aws.default"
@@ -169,9 +169,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.11.3
-ip-10-0-19-112   Ready     34m       v1.11.3
-ip-10-0-4-22     Ready     34m       v1.11.3
+ip-10-0-12-221   Ready     34m       v1.12.2
+ip-10-0-19-112   Ready     34m       v1.12.2
+ip-10-0-4-22     Ready     34m       v1.12.2
 ```
 
 List the pods.
@@ -197,7 +197,7 @@ kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance) and [addons](/addons/overview).
 
 !!! note
     On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@@ -244,9 +244,10 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| disk_iops | IOPS of the EBS volume | "0" (i.e. auto) | "400" |
 | worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization.md) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization.md) |
+| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
 | host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |

docs/cl/azure.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on Azure with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on Azure with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -40,7 +40,7 @@ providers {
 }
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -58,7 +58,7 @@ Configure the Azure provider in a `providers.tf` file.
 
 ```tf
 provider "azurerm" {
-  version = "1.13.0"
+  version = "1.16.0"
   alias   = "default"
 }
 
@@ -91,7 +91,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.
 
 ```tf
 module "azure-ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.12.2"
 
   providers = {
     azurerm  = "azurerm.default"
@@ -165,10 +165,10 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/ramius/auth/kubeconfig
 $ kubectl get nodes
 NAME                  STATUS  ROLES              AGE  VERSION
-ramius-controller-0   Ready   controller,master  24m  v1.11.3
-ramius-worker-000001  Ready   node               25m  v1.11.3
-ramius-worker-000002  Ready   node               24m  v1.11.3
-ramius-worker-000005  Ready   node               24m  v1.11.3
+ramius-controller-0   Ready   controller,master  24m  v1.12.2
+ramius-worker-000001  Ready   node               25m  v1.12.2
+ramius-worker-000002  Ready   node               24m  v1.12.2
+ramius-worker-000005  Ready   node               24m  v1.12.2
 ```
 
 List the pods.
@@ -196,7 +196,7 @@ kube-system   pod-checkpointer-cnqdg-ramius-controller-0  1/1    Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 !!! note
     On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.

docs/cl/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.11.3 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.12.2 cluster on bare-metal with Container Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -91,7 +91,7 @@ For networks already supporting iPXE clients, you can add a `default.ipxe` confi
 chain http://matchbox.foo:8080/boot.ipxe
 ```
 
-For networks with Ubiquiti Routers, you can [configure the router](/topics/hardware.md#ubiquiti) itself to chainload machines to iPXE and Matchbox.
+For networks with Ubiquiti Routers, you can [configure the router](/topics/hardware/#ubiquiti) itself to chainload machines to iPXE and Matchbox.
 
 For a small lab, you may wish to checkout the [quay.io/coreos/dnsmasq](https://quay.io/repository/coreos/dnsmasq) container image and [copy-paste examples](https://github.com/coreos/matchbox/blob/master/Documentation/network-setup.md#coreosdnsmasq).
 
@@ -137,7 +137,7 @@ providers {
 }
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -182,7 +182,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.12.2"
   
   providers = {
     local = "local.default"
@@ -291,9 +291,9 @@ Apply complete! Resources: 55 added, 0 changed, 0 destroyed.
 To watch the install to disk (until machines reboot from disk), SSH to port 2222.
 
 ```
-# before v1.11.3
+# before v1.12.2
 $ ssh debug@node1.example.com
-# after v1.11.3
+# after v1.12.2
 $ ssh -p 2222 core@node1.example.com
 ```
 
@@ -318,9 +318,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.11.3
-node2.example.com   Ready     11m       v1.11.3
-node3.example.com   Ready     11m       v1.11.3
+node1.example.com   Ready     11m       v1.12.2
+node2.example.com   Ready     11m       v1.12.2
+node3.example.com   Ready     11m       v1.12.2
 ```
 
 List the pods.
@@ -346,7 +346,7 @@ kube-system   pod-checkpointer-wf65d-node1.example.com   1/1       Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 !!! note
     On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@@ -377,7 +377,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
-| cached_install | Whether machines should PXE boot and install from the Matchbox `/assets` cache. Admin MUST have downloaded Container Linux images into the cache to use this (coreos only for now) | false | true |
+| cached_install | PXE boot and install from the Matchbox `/assets` cache. Admin MUST have downloaded Container Linux or Flatcar images into the cache | false | true |
 | install_disk | Disk device where Container Linux should be installed | "/dev/sda" | "/dev/sdb" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico-only) | 1480 | - | 

docs/cl/digital-ocean.md

@@ -1,6 +1,6 @@
 # Digital Ocean
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on DigitalOcean with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -37,7 +37,7 @@ providers {
 }
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -58,7 +58,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.
 
 ```tf
 provider "digitalocean" {
-  version = "0.1.3"
+  version = "1.0.0"
   token = "${chomp(file("~/.config/digital-ocean/token"))}"
   alias = "default"
 }
@@ -90,7 +90,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.12.2"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -164,9 +164,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.11.3
-10.132.115.81    Ready     10m       v1.11.3
-10.132.124.107   Ready     10m       v1.11.3
+10.132.110.130   Ready     10m       v1.12.2
+10.132.115.81    Ready     10m       v1.12.2
+10.132.124.107   Ready     10m       v1.12.2
 ```
 
 List the pods.
@@ -191,7 +191,7 @@ kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 !!! note
     On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@@ -254,8 +254,8 @@ Digital Ocean requires the SSH public key be uploaded to your account, so you ma
 | controller_type | Droplet type for controllers | s-2vcpu-2gb | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
 | worker_type | Droplet type for workers | s-1vcpu-1gb | s-1vcpu-1gb, s-1vcpu-2gb, s-2vcpu-2gb, ... |
 | image | Container Linux image for instances | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advnaced/customization.md) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](customization.md) |
+| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | cluster_domain_suffix | FQDN suffix for Kubernetes services answered by coredns. | "cluster.local" | "k8s.example.com" |

docs/cl/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.11.3 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.12.2 cluster on Google Compute Engine with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -37,7 +37,7 @@ providers {
 }
 ```
 
-Read [concepts](/architecture/concepts.md) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
 
 ```
 cd infra/clusters
@@ -97,7 +97,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
   
   providers = {
     google   = "google.default"
@@ -172,9 +172,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.11.3
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.11.3
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.11.3
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
 ```
 
 List the pods.
@@ -199,7 +199,7 @@ kube-system   pod-checkpointer-l6lrt                    1/1    Running   0
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance.md) and [addons](/addons/overview.md).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 !!! note
     On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@@ -249,8 +249,8 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | os_image | Container Linux image for compute instances | "coreos-stable" | "coreos-stable-1632-3-0-v20180215" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization.md) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](customization.md) |
+| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
@@ -46,7 +46,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
   
   providers = {
     google   = "google.default"
@@ -87,9 +87,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.11.3
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.11.3
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.11.3
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
 ```
 
 List the pods.

docs/topics/maintenance.md

@@ -18,7 +18,7 @@ module "google-cloud-yavin" {
 }
 
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.11.3"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.12.2"
   ...
 }
 ```
@@ -40,7 +40,7 @@ Blue-green replacement reduces risk for clusters running critical applications.
 Blue-green replacement provides some subtler benefits as well:
 
 * Encourages investment in tooling for traffic migration and failovers. When a cluster incident arises, shifting applications to a healthy cluster will be second nature.
-* Discourages reliance on in-place opqaue state. Retain confidence in your ability to create infrastructure from scratch.
+* Discourages reliance on in-place opaque state. Retain confidence in your ability to create infrastructure from scratch.
 * Allows Typhoon to make architecture changes between releases and eases the burden on Typhoon maintainers. By contrast, distros promising in-place upgrades get stuck with their mistakes or require complex and error-prone migrations.
 
 ### Bare-Metal

docs/topics/security.md

@@ -10,7 +10,7 @@ Typhoon aims to be minimal and secure. We're running it ourselves after all.
 * Generated kubelet TLS certificates and `kubeconfig` (365 days)
 * [Role-Based Access Control](https://kubernetes.io/docs/admin/authorization/rbac/) is enabled. Apps must define RBAC policies
 * Workloads run on worker nodes only, unless they tolerate the master taint
-* Kubernetes [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) and Calico [Policy](https://docs.projectcalico.org/latest/reference/calicoctl/resources/policy) support [^1]
+* Kubernetes [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) and Calico [NetworkPolicy](https://docs.projectcalico.org/latest/reference/calicoctl/resources/networkpolicy) support [^1]
 
 [^1]: Requires `networking = "calico"`. Calico is the default on AWS, bare-metal, and Google Cloud. Azure and Digital Ocean are limited to `networking = "flannel"`.
 

google-cloud/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

google-cloud/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -90,6 +90,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -123,7 +124,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.3
+          KUBELET_IMAGE_TAG=v1.12.2
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

google-cloud/container-linux/kubernetes/controllers.tf

@@ -57,6 +57,12 @@ resource "google_compute_instance" "controllers" {
 
   can_ip_forward = true
   tags           = ["${var.cluster_name}-controller"]
+
+  lifecycle {
+    ignore_changes = [
+      "metadata",
+    ]
+  }
 }
 
 # Controller Ignition configs

google-cloud/container-linux/kubernetes/ingress.tf

@@ -1,13 +1,19 @@
-# Static IPv4 address for the TCP Proxy Load Balancer
+# Static IPv4 address for Ingress Load Balancing
 resource "google_compute_global_address" "ingress-ipv4" {
-  name       = "${var.cluster_name}-ingress-ip"
+  name       = "${var.cluster_name}-ingress-ipv4"
   ip_version = "IPV4"
 }
 
+# Static IPv6 address for Ingress Load Balancing
+resource "google_compute_global_address" "ingress-ipv6" {
+  name       = "${var.cluster_name}-ingress-ipv6"
+  ip_version = "IPV6"
+}
+
 # Forward IPv4 TCP traffic to the HTTP proxy load balancer
 # Google Cloud does not allow TCP proxies for port 80. Must use HTTP proxy.
-resource "google_compute_global_forwarding_rule" "ingress-http" {
-  name        = "${var.cluster_name}-ingress-http"
+resource "google_compute_global_forwarding_rule" "ingress-http-ipv4" {
+  name        = "${var.cluster_name}-ingress-http-ipv4"
   ip_address  = "${google_compute_global_address.ingress-ipv4.address}"
   ip_protocol = "TCP"
   port_range  = "80"
@@ -15,14 +21,33 @@ resource "google_compute_global_forwarding_rule" "ingress-http" {
 }
 
 # Forward IPv4 TCP traffic to the TCP proxy load balancer
-resource "google_compute_global_forwarding_rule" "ingress-https" {
-  name        = "${var.cluster_name}-ingress-https"
+resource "google_compute_global_forwarding_rule" "ingress-https-ipv4" {
+  name        = "${var.cluster_name}-ingress-https-ipv4"
   ip_address  = "${google_compute_global_address.ingress-ipv4.address}"
   ip_protocol = "TCP"
   port_range  = "443"
   target      = "${google_compute_target_tcp_proxy.ingress-https.self_link}"
 }
 
+# Forward IPv6 TCP traffic to the HTTP proxy load balancer
+# Google Cloud does not allow TCP proxies for port 80. Must use HTTP proxy.
+resource "google_compute_global_forwarding_rule" "ingress-http-ipv6" {
+  name        = "${var.cluster_name}-ingress-http-ipv6"
+  ip_address  = "${google_compute_global_address.ingress-ipv6.address}"
+  ip_protocol = "TCP"
+  port_range  = "80"
+  target      = "${google_compute_target_http_proxy.ingress-http.self_link}"
+}
+
+# Forward IPv6 TCP traffic to the TCP proxy load balancer
+resource "google_compute_global_forwarding_rule" "ingress-https-ipv6" {
+  name        = "${var.cluster_name}-ingress-https-ipv6"
+  ip_address  = "${google_compute_global_address.ingress-ipv6.address}"
+  ip_protocol = "TCP"
+  port_range  = "443"
+  target      = "${google_compute_target_tcp_proxy.ingress-https.self_link}"
+}
+
 # HTTP proxy load balancer for ingress controllers
 resource "google_compute_target_http_proxy" "ingress-http" {
   name        = "${var.cluster_name}-ingress-http"

google-cloud/container-linux/kubernetes/network.tf

@@ -123,20 +123,6 @@ resource "google_compute_firewall" "internal-kubelet" {
   target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "google_compute_firewall" "internal-kubelet-readonly" {
-  name    = "${var.cluster_name}-internal-kubelet-readonly"
-  network = "${google_compute_network.network.name}"
-
-  allow {
-    protocol = "tcp"
-    ports    = [10255]
-  }
-
-  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
-  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
-}
-
 # Workers
 
 resource "google_compute_firewall" "allow-ingress" {