Update mkdocs-material docs theme version

Update Grafana from v5.3.1 to v5.3.2
* https://github.com/grafana/grafana/releases/tag/v5.3.2
2025-08-03 12:31:33 +02:00 · 2018-10-28 19:45:58 -07:00 · 2018-10-28 19:44:22 -07:00 · 2018-10-28 16:48:12 -07:00 · 2018-10-28 14:57:31 -07:00 · 2018-10-28 14:30:58 -07:00
140 changed files with 3045 additions and 635 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@ -4,11 +4,11 @@

 ### Environment

-* Platform: aws, bare-metal, google-cloud, digital-ocean
+* Platform: aws, azure, bare-metal, google-cloud, digital-ocean
 * OS: container-linux, fedora-atomic
-* Terraform: `terraform version`
-* Plugins: Provider plugin versions
-* Ref: Git SHA (if applicable)
+* Ref: Release version or Git SHA (reporting latest is **not** helpful)
+* Terraform: `terraform version` (reporting latest is **not** helpful)
+* Plugins: Provider plugin versions (reporting latest is **not** helpful)

 ### Problem

--- a/CHANGES.md
+++ b/CHANGES.md
@ -2,6 +2,105 @@

 Notable changes between versions.

+## Latest
+
+## v1.12.2
+
+* Kubernetes [v1.12.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122)
+* Update CoreDNS from 1.2.2 to [1.2.4](https://github.com/coredns/coredns/releases/tag/v1.2.4)
+* Update Calico from v3.2.3 to [v3.3.0](https://docs.projectcalico.org/v3.3/releases/)
+* Disable Kubelet read-only port ([#324](https://github.com/poseidon/typhoon/pull/324))
+* Fix CoreDNS AntiAffinity spec to prefer spreading replicas
+* Ignore controller node user-data changes ([#335](https://github.com/poseidon/typhoon/pull/335))
+  * Once all managed clusters use v1.12.2, it is possible to update `terraform-provider-ct`
+
+#### AWS
+
+* Add `disk_iops` variable for EBS volume IOPS ([#314](https://github.com/poseidon/typhoon/pull/314))
+
+#### Azure
+
+* Use new `azurerm_network_interface_backend_address_pool_association` ([#332](https://github.com/poseidon/typhoon/pull/332))
+  * Require `terraform-provider-azurerm` v1.17+ (action required)
+* Add `primary` field to `ip_configuration` needed by v1.17+ ([#331](https://github.com/poseidon/typhoon/pull/331))
+
+#### DigitalOcean
+
+* Add AAAA DNS records resolving to worker nodes ([#333](https://github.com/poseidon/typhoon/pull/333))
+  * Hosting IPv6 apps requires editing nginx-ingress with `hostNetwork: true`
+
+#### Google Cloud
+
+* Add an IPv6 address and IPv6 forwarding rules for load balancing IPv6 Ingress ([#334](https://github.com/poseidon/typhoon/pull/334))
+  * Add `ingress_static_ipv6` output variable for use in AAAA DNS records
+  * Allow serving IPv6 applications via Kubernetes Ingress
+
+#### Addons
+
+* Configure Heapster to scrape Kubelets with bearer token auth ([#323](https://github.com/poseidon/typhoon/pull/323))
+* Update Grafana from v5.3.1 to v5.3.2
+
+## v1.12.1
+
+* Kubernetes [v1.12.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1121)
+* Update etcd from v3.3.9 to [v3.3.10](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10)
+* Update CoreDNS from 1.1.3 to [1.2.2](https://github.com/coredns/coredns/releases/tag/v1.2.2)
+* Update Calico from v3.2.1 to [v3.2.3](https://docs.projectcalico.org/v3.2/releases/)
+* Raise scheduler and controller-manager replicas to the larger of 2 or the number of controller nodes ([#312](https://github.com/poseidon/typhoon/pull/312))
+  * Single-controller clusters continue to run 2 replicas as before
+* Raise default CoreDNS replicas to the larger of 2 or the number of controller nodes ([#313](https://github.com/poseidon/typhoon/pull/313))
+  * Add AntiAffinity preferred rule to favor spreading CoreDNS pods
+* Annotate control plane and addon containers to use the Docker runtime seccomp profile ([#319](https://github.com/poseidon/typhoon/pull/319))
+  * Override Kubernetes default behavior that starts containers with `seccomp=unconfined`
+
+#### Azure
+
+* Remove `admin_password` field (disabled) since it is now optional
+  * Require `terraform-provider-azurerm` v1.16+ (action required)
+
+#### Bare-Metal
+
+* Add support for `cached_install` mode with Flatcar Linux ([#315](https://github.com/poseidon/typhoon/pull/315))
+
+#### DigitalOcean
+
+* Require `terraform-provider-digitalocean` v1.0+ (action required)
+
+#### Addons
+
+* Update nginx-ingress from v0.19.0 to v0.20.0
+* Update Prometheus from v2.3.2 to v2.4.3
+* Update Grafana from v5.2.4 to v5.3.1
+
+## v1.11.3
+
+* Kubernetes [v1.11.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113)
+* Introduce Typhoon for Azure as alpha ([#288](https://github.com/poseidon/typhoon/pull/288))
+  * Special thanks @justaugustus for an earlier variant
+* Update Calico from v3.1.3 to v3.2.1 ([#278](https://github.com/poseidon/typhoon/pull/278))
+
+#### AWS
+
+* Remove firewall rule allowing ICMP packets to nodes ([#285](https://github.com/poseidon/typhoon/pull/285))
+
+#### Bare-Metal
+
+* Remove `controller_networkds` and `worker_networkds` variables. Use Container Linux Config snippets [#277](https://github.com/poseidon/typhoon/pull/277)
+
+#### Google Cloud
+
+* Fix firewall to allow etcd client port 2379 traffic between controller nodes ([#287](https://github.com/poseidon/typhoon/pull/287))
+  * kube-apiservers were only able to connect to their node's local etcd peer. While master node outages were tolerated, reaching a healthy peer took longer than neccessary in some cases
+  * Reduce time needed to bootstrap the cluster
+* Remove firewall rule allowing workers to access Nginx Ingress health check ([#284](https://github.com/poseidon/typhoon/pull/284))
+  * Nginx Ingress addon no longer uses hostNetwork, Prometheus scrapes via CNI network
+
+#### Addons
+
+* Update nginx-ingress from 0.17.1 to 0.19.0
+* Update kube-state-metrics from v1.3.1 to v1.4.0
+* Update Grafana from 5.2.2 to 5.2.4
+
 ## v1.11.2

 * Kubernetes [v1.11.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1112)
@ -14,7 +113,7 @@ Notable changes between versions.
 * Introduce [Container Linux Config snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) on bare-metal
  * Validate and additively merge custom Container Linux Configs during terraform plan
  * Define files, systemd units, dropins, networkd configs, mounts, users, and more
-  * [Require](https://typhoon.psdn.io/cl/bare-metal/#terraform-setup) `terraform-provider-ct` plugin v0.2.1 (action required!)
+  * [Require](https://typhoon.psdn.io/cl/bare-metal/#terraform-setup) `terraform-provider-ct` plugin v0.2.1 (**action required!**)

 #### Addons

--- a/README.md
+++ b/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/google-cloud/#preemption) (varies by platform)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Modules
@ -25,6 +25,7 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 |---------------|------------------|------------------|--------|
 | AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
+| Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
 | Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
@ -38,7 +39,7 @@ The AWS and bare-metal `container-linux` modules allow picking Red Hat Container

 * [Docs](https://typhoon.psdn.io)
 * Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
-* Tutorials for [AWS](https://typhoon.psdn.io/cl/aws/), [Bare-Metal](https://typhoon.psdn.io/cl/bare-metal/), [Digital Ocean](https://typhoon.psdn.io/cl/digital-ocean/), and [Google-Cloud](https://typhoon.psdn.io/cl/google-cloud/)
+* Tutorials for [AWS](docs/cl/aws.md), [Azure](docs/cl/azure.md), [Bare-Metal](docs/cl/bare-metal.md), [Digital Ocean](docs/cl/digital-ocean.md), and [Google-Cloud](docs/cl/google-cloud.md)

 ## Usage

@ -46,7 +47,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.11.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
  
  providers = {
    google   = "google.default"
@ -71,15 +72,14 @@ module "google-cloud-yavin" {
 }
 ```

-Fetch modules, plan the changes to be made, and apply the changes.
+Initialize modules, plan the changes to be made, and apply the changes.

 ```sh
 $ terraform init
-$ terraform get --update
 $ terraform plan
-Plan: 37 to add, 0 to change, 0 to destroy.
+Plan: 64 to add, 0 to change, 0 to destroy.
 $ terraform apply
-Apply complete! Resources: 37 added, 0 changed, 0 destroyed.
+Apply complete! Resources: 64 added, 0 changed, 0 destroyed.
 ```

 In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.
@ -88,9 +88,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.11.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.11.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.11.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
 ```

 List the pods.
--- a/addons/cluo/update-agent.yaml
+++ b/addons/cluo/update-agent.yaml
@ -15,6 +15,8 @@ spec:
    metadata:
      labels:
        app: container-linux-update-agent
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
      - name: update-agent
--- a/addons/cluo/update-operator.yaml
+++ b/addons/cluo/update-operator.yaml
@ -12,6 +12,8 @@ spec:
    metadata:
      labels:
        app: container-linux-update-operator
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
      - name: update-operator
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -18,10 +18,12 @@ spec:
      labels:
        name: grafana
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
        - name: grafana
-          image: grafana/grafana:5.2.2
+          image: grafana/grafana:5.3.2
          env:
            - name: GF_SERVER_HTTP_PORT
              value: "8080"
--- a/addons/heapster/cluster-role-binding.yaml
+++ b/addons/heapster/cluster-role-binding.yaml
@ -5,7 +5,7 @@ metadata:
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
-  name: system:heapster
+  name: heapster
 subjects:
 - kind: ServiceAccount
  name: heapster
--- a/addons/heapster/cluster-role.yaml
+++ b/addons/heapster/cluster-role.yaml
@ -0,0 +1,30 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: heapster
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - events
+  - namespaces
+  - nodes
+  - pods
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - deployments
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - nodes/stats
+  verbs:
+  - get
--- a/addons/heapster/deployment.yaml
+++ b/addons/heapster/deployment.yaml
@ -23,7 +23,7 @@ spec:
          image: k8s.gcr.io/heapster-amd64:v1.5.4
          command:
            - /heapster
-            - --source=kubernetes.summary_api:''
+            - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
          livenessProbe:
            httpGet:
              path: /healthz
--- a/addons/nginx-ingress/aws/default-backend/deployment.yaml
+++ b/addons/nginx-ingress/aws/default-backend/deployment.yaml
@ -14,6 +14,8 @@ spec:
      labels:
        name: default-backend
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
        - name: default-backend
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -17,12 +17,14 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      nodeSelector:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/azure/0-namespace.yaml
+++ b/addons/nginx-ingress/azure/0-namespace.yaml
@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ingress
+  labels:
+    name: ingress
--- a/addons/nginx-ingress/azure/default-backend/deployment.yaml
+++ b/addons/nginx-ingress/azure/default-backend/deployment.yaml
@ -0,0 +1,42 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: default-backend
+  namespace: ingress
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: default-backend
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: default-backend
+        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
+    spec:
+      containers:
+        - name: default-backend
+          # Any image is permissable as long as:
+          # 1. It serves a 404 page at /
+          # 2. It serves 200 on a /healthz endpoint
+          image: k8s.gcr.io/defaultbackend:1.4
+          ports:
+            - containerPort: 8080
+          resources:
+            limits:
+              cpu: 10m
+              memory: 20Mi
+            requests:
+              cpu: 10m
+              memory: 20Mi
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+              scheme: HTTP
+            initialDelaySeconds: 30
+            timeoutSeconds: 5
+      terminationGracePeriodSeconds: 60
--- a/addons/nginx-ingress/azure/default-backend/service.yaml
+++ b/addons/nginx-ingress/azure/default-backend/service.yaml
@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: default-backend
+  namespace: ingress
+spec:
+  type: ClusterIP
+  selector:
+    name: default-backend
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 8080
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -0,0 +1,79 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nginx-ingress-controller
+  namespace: ingress
+spec:
+  replicas: 2
+  strategy:
+    rollingUpdate:
+      maxUnavailable: 1
+  selector:
+    matchLabels:
+      name: nginx-ingress-controller
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: nginx-ingress-controller
+        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
+    spec:
+      nodeSelector:
+        node-role.kubernetes.io/node: ""
+      containers:
+        - name: nginx-ingress-controller
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          args:
+            - /nginx-ingress-controller
+            - --default-backend-service=$(POD_NAMESPACE)/default-backend
+            - --ingress-class=public
+          # use downward API
+          env:
+            - name: POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          ports:
+            - name: http
+              containerPort: 80
+              hostPort: 80
+            - name: https
+              containerPort: 443
+              hostPort: 443
+            - name: health
+              containerPort: 10254
+              hostPort: 10254
+          livenessProbe:
+            failureThreshold: 3
+            httpGet:
+              path: /healthz
+              port: 10254
+              scheme: HTTP
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            successThreshold: 1
+            timeoutSeconds: 1
+          readinessProbe:
+            failureThreshold: 3
+            httpGet:
+              path: /healthz
+              port: 10254
+              scheme: HTTP
+            periodSeconds: 10
+            successThreshold: 1
+            timeoutSeconds: 1
+          securityContext:
+            capabilities:
+              add:
+              - NET_BIND_SERVICE
+              drop:
+              - ALL
+            runAsUser: 33 # www-data
+      restartPolicy: Always
+      terminationGracePeriodSeconds: 60
--- a/addons/nginx-ingress/azure/rbac/cluster-role-binding.yaml
+++ b/addons/nginx-ingress/azure/rbac/cluster-role-binding.yaml
@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: ingress
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: ingress
+subjects:
+  - kind: ServiceAccount
+    namespace: ingress
+    name: default
--- a/addons/nginx-ingress/azure/rbac/cluster-role.yaml
+++ b/addons/nginx-ingress/azure/rbac/cluster-role.yaml
@ -0,0 +1,51 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: ingress
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - endpoints
+      - nodes
+      - pods
+      - secrets
+    verbs:
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - services
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+        - events
+    verbs:
+        - create
+        - patch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses/status
+    verbs:
+      - update
--- a/addons/nginx-ingress/azure/rbac/role-binding.yaml
+++ b/addons/nginx-ingress/azure/rbac/role-binding.yaml
@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ingress
+  namespace: ingress
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ingress
+subjects:
+  - kind: ServiceAccount
+    namespace: ingress
+    name: default
--- a/addons/nginx-ingress/azure/rbac/role.yaml
+++ b/addons/nginx-ingress/azure/rbac/role.yaml
@ -0,0 +1,41 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ingress
+  namespace: ingress
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - pods
+      - secrets
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    resourceNames:
+      # Defaults to "<election-id>-<ingress-class>"
+      # Here: "<ingress-controller-leader>-<nginx>"
+      # This has to be adapted if you change either parameter
+      # when launching the nginx-ingress-controller.
+      - "ingress-controller-leader-public"
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - endpoints
+    verbs:
+      - get
+      - create
+      - update
--- a/addons/nginx-ingress/azure/service.yaml
+++ b/addons/nginx-ingress/azure/service.yaml
@ -0,0 +1,22 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: nginx-ingress-controller
+  namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
+spec:
+  type: ClusterIP
+  selector:
+    name: nginx-ingress-controller
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 80
+    - name: https
+      protocol: TCP
+      port: 443
+      targetPort: 443
--- a/addons/nginx-ingress/bare-metal/default-backend/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/default-backend/deployment.yaml
@ -14,6 +14,8 @@ spec:
      labels:
        name: default-backend
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
        - name: default-backend
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -17,10 +17,12 @@ spec:
      labels:
        name: ingress-controller-public
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -17,12 +17,14 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      nodeSelector:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/digital-ocean/default-backend/deployment.yaml
+++ b/addons/nginx-ingress/digital-ocean/default-backend/deployment.yaml
@ -14,6 +14,8 @@ spec:
      labels:
        name: default-backend
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
        - name: default-backend
--- a/addons/nginx-ingress/google-cloud/default-backend/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/default-backend/deployment.yaml
@ -14,6 +14,8 @@ spec:
      labels:
        name: default-backend
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
        - name: default-backend
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -17,12 +17,14 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      nodeSelector:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -14,11 +14,13 @@ spec:
      labels:
        name: prometheus
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.3.2
+          image: quay.io/prometheus/prometheus:v2.4.3
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -18,11 +18,13 @@ spec:
      labels:
        name: kube-state-metrics
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.3.1
+        image: quay.io/coreos/kube-state-metrics:v1.4.0
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -17,6 +17,8 @@ spec:
      labels:
        name: node-exporter
        phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      serviceAccountName: node-exporter
      securityContext:
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -299,7 +299,7 @@ data:
        labels:
          severity: warning
        annotations:
-          description: Pod {{$labels.namespaces}}/{{$labels.pod}} is was restarted {{$value}}
+          description: Pod {{$labels.namespaces}}/{{$labels.pod}} restarted {{$value}}
            times within the last hour
          summary: Pod is restarting frequently
  kubelet.rules.yaml: |
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
--- a/aws/container-linux/kubernetes/bootkube.tf
+++ b/aws/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=70c28399703cb4ec8930394682400d90d733e5a5"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -88,6 +88,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -122,7 +123,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.2
+          KUBELET_IMAGE_TAG=v1.12.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -24,12 +24,13 @@ resource "aws_instance" "controllers" {
  instance_type = "${var.controller_type}"

  ami       = "${local.ami_id}"
-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"

  # storage
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
@ -38,12 +39,23 @@ resource "aws_instance" "controllers" {
  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]

  lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
  }
 }

-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
  count = "${var.controller_count}"

  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@ -73,10 +85,3 @@ data "template_file" "etcds" {
    dns_zone     = "${var.dns_zone}"
  }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
-}
--- a/aws/container-linux/kubernetes/nlb.tf
+++ b/aws/container-linux/kubernetes/nlb.tf
@ -5,7 +5,7 @@ resource "aws_route53_record" "apiserver" {
  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
  type = "A"

-  # AWS recommends their special "alias" records for ELBs
+  # AWS recommends their special "alias" records for NLBs
  alias {
    name                   = "${aws_lb.nlb.dns_name}"
    zone_id                = "${aws_lb.nlb.zone_id}"
--- a/aws/container-linux/kubernetes/security.tf
+++ b/aws/container-linux/kubernetes/security.tf
@ -11,16 +11,6 @@ resource "aws_security_group" "controller" {
  tags = "${map("Name", "${var.cluster_name}-controller")}"
 }

-resource "aws_security_group_rule" "controller-icmp" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "icmp"
-  from_port   = 0
-  to_port     = 0
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "controller-ssh" {
  security_group_id = "${aws_security_group.controller.id}"

@ -31,16 +21,6 @@ resource "aws_security_group_rule" "controller-ssh" {
  cidr_blocks = ["0.0.0.0/0"]
 }

-resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 6443
-  to_port     = 6443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "controller-etcd" {
  security_group_id = "${aws_security_group.controller.id}"

@ -51,6 +31,7 @@ resource "aws_security_group_rule" "controller-etcd" {
  self      = true
 }

+# Allow Prometheus to scrape etcd metrics
 resource "aws_security_group_rule" "controller-etcd-metrics" {
  security_group_id = "${aws_security_group.controller.id}"

@ -61,6 +42,16 @@ resource "aws_security_group_rule" "controller-etcd-metrics" {
  source_security_group_id = "${aws_security_group.worker.id}"
 }

+resource "aws_security_group_rule" "controller-apiserver" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 6443
+  to_port     = 6443
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
 resource "aws_security_group_rule" "controller-flannel" {
  security_group_id = "${aws_security_group.controller.id}"

@ -81,6 +72,7 @@ resource "aws_security_group_rule" "controller-flannel-self" {
  self      = true
 }

+# Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "controller-node-exporter" {
  security_group_id = "${aws_security_group.controller.id}"

@ -91,6 +83,7 @@ resource "aws_security_group_rule" "controller-node-exporter" {
  source_security_group_id = "${aws_security_group.worker.id}"
 }

+# Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "controller-kubelet" {
  security_group_id = "${aws_security_group.controller.id}"

@ -111,26 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
  self      = true
 }

-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
  security_group_id = "${aws_security_group.controller.id}"

@ -213,16 +186,6 @@ resource "aws_security_group" "worker" {
  tags = "${map("Name", "${var.cluster_name}-worker")}"
 }

-resource "aws_security_group_rule" "worker-icmp" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "icmp"
-  from_port   = 0
-  to_port     = 0
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "worker-ssh" {
  security_group_id = "${aws_security_group.worker.id}"

@ -273,6 +236,7 @@ resource "aws_security_group_rule" "worker-flannel-self" {
  self      = true
 }

+# Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "worker-node-exporter" {
  security_group_id = "${aws_security_group.worker.id}"

@ -293,6 +257,7 @@ resource "aws_security_group_rule" "ingress-health" {
  cidr_blocks = ["0.0.0.0/0"]
 }

+# Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "worker-kubelet" {
  security_group_id = "${aws_security_group.worker.id}"

@ -303,6 +268,7 @@ resource "aws_security_group_rule" "worker-kubelet" {
  source_security_group_id = "${aws_security_group.controller.id}"
 }

+# Allow Prometheus to scrape kubelet metrics
 resource "aws_security_group_rule" "worker-kubelet-self" {
  security_group_id = "${aws_security_group.worker.id}"

@ -313,26 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
  self      = true
 }

-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
  security_group_id = "${aws_security_group.worker.id}"

--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -59,6 +59,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
  type        = "string"
  default     = ""
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -60,6 +60,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -92,7 +93,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.2
+          KUBELET_IMAGE_TAG=v1.12.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -110,7 +111,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.11.2 \
+            docker://k8s.gcr.io/hyperkube:v1.12.2 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -52,6 +52,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
  type        = "string"
  default     = ""
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -46,12 +46,13 @@ resource "aws_launch_configuration" "worker" {
  spot_price        = "${var.spot_price}"
  enable_monitoring = false

-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"

  # storage
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
@ -64,8 +65,15 @@ resource "aws_launch_configuration" "worker" {
  }
 }

-# Worker Container Linux Config
-data "template_file" "worker_config" {
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.clc_snippets}"]
+}
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars = {
@ -75,9 +83,3 @@ data "template_file" "worker_config" {
    cluster_domain_suffix = "${var.cluster_domain_suffix}"
  }
 }
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
-  pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
-}
--- a/aws/fedora-atomic/kubernetes/README.md
+++ b/aws/fedora-atomic/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
--- a/aws/fedora-atomic/kubernetes/bootkube.tf
+++ b/aws/fedora-atomic/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=70c28399703cb4ec8930394682400d90d733e5a5"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -65,6 +65,7 @@ write_files:
        --node-labels=node-role.kubernetes.io/master \
        --node-labels=node-role.kubernetes.io/controller="true" \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/kubernetes/kubeconfig
@ -92,8 +93,8 @@ bootcmd:
 runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
  - [systemctl, start, --no-block, etcd.service]
  - [systemctl, enable, cloud-metadata.service]
--- a/aws/fedora-atomic/kubernetes/controllers.tf
+++ b/aws/fedora-atomic/kubernetes/controllers.tf
@ -30,6 +30,7 @@ resource "aws_instance" "controllers" {
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
@ -38,7 +39,10 @@ resource "aws_instance" "controllers" {
  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]

  lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
  }
 }

--- a/aws/fedora-atomic/kubernetes/nlb.tf
+++ b/aws/fedora-atomic/kubernetes/nlb.tf
@ -5,7 +5,7 @@ resource "aws_route53_record" "apiserver" {
  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
  type = "A"

-  # AWS recommends their special "alias" records for ELBs
+  # AWS recommends their special "alias" records for NLBs
  alias {
    name                   = "${aws_lb.nlb.dns_name}"
    zone_id                = "${aws_lb.nlb.zone_id}"
--- a/aws/fedora-atomic/kubernetes/security.tf
+++ b/aws/fedora-atomic/kubernetes/security.tf
@ -11,16 +11,6 @@ resource "aws_security_group" "controller" {
  tags = "${map("Name", "${var.cluster_name}-controller")}"
 }

-resource "aws_security_group_rule" "controller-icmp" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "icmp"
-  from_port   = 0
-  to_port     = 0
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "controller-ssh" {
  security_group_id = "${aws_security_group.controller.id}"

@ -31,16 +21,6 @@ resource "aws_security_group_rule" "controller-ssh" {
  cidr_blocks = ["0.0.0.0/0"]
 }

-resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 6443
-  to_port     = 6443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "controller-etcd" {
  security_group_id = "${aws_security_group.controller.id}"

@ -51,6 +31,7 @@ resource "aws_security_group_rule" "controller-etcd" {
  self      = true
 }

+# Allow Prometheus to scrape etcd metrics
 resource "aws_security_group_rule" "controller-etcd-metrics" {
  security_group_id = "${aws_security_group.controller.id}"

@ -61,6 +42,16 @@ resource "aws_security_group_rule" "controller-etcd-metrics" {
  source_security_group_id = "${aws_security_group.worker.id}"
 }

+resource "aws_security_group_rule" "controller-apiserver" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 6443
+  to_port     = 6443
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
 resource "aws_security_group_rule" "controller-flannel" {
  security_group_id = "${aws_security_group.controller.id}"

@ -81,6 +72,7 @@ resource "aws_security_group_rule" "controller-flannel-self" {
  self      = true
 }

+# Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "controller-node-exporter" {
  security_group_id = "${aws_security_group.controller.id}"

@ -91,6 +83,7 @@ resource "aws_security_group_rule" "controller-node-exporter" {
  source_security_group_id = "${aws_security_group.worker.id}"
 }

+# Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "controller-kubelet" {
  security_group_id = "${aws_security_group.controller.id}"

@ -111,26 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
  self      = true
 }

-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
  security_group_id = "${aws_security_group.controller.id}"

@ -213,16 +186,6 @@ resource "aws_security_group" "worker" {
  tags = "${map("Name", "${var.cluster_name}-worker")}"
 }

-resource "aws_security_group_rule" "worker-icmp" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "icmp"
-  from_port   = 0
-  to_port     = 0
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "worker-ssh" {
  security_group_id = "${aws_security_group.worker.id}"

@ -273,6 +236,7 @@ resource "aws_security_group_rule" "worker-flannel-self" {
  self      = true
 }

+# Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "worker-node-exporter" {
  security_group_id = "${aws_security_group.worker.id}"

@ -293,6 +257,7 @@ resource "aws_security_group_rule" "ingress-health" {
  cidr_blocks = ["0.0.0.0/0"]
 }

+# Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "worker-kubelet" {
  security_group_id = "${aws_security_group.worker.id}"

@ -303,6 +268,7 @@ resource "aws_security_group_rule" "worker-kubelet" {
  source_security_group_id = "${aws_security_group.controller.id}"
 }

+# Allow Prometheus to scrape kubelet metrics
 resource "aws_security_group_rule" "worker-kubelet-self" {
  security_group_id = "${aws_security_group.worker.id}"

@ -313,26 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
  self      = true
 }

-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
  security_group_id = "${aws_security_group.worker.id}"

--- a/aws/fedora-atomic/kubernetes/variables.tf
+++ b/aws/fedora-atomic/kubernetes/variables.tf
@ -53,6 +53,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
  type        = "string"
  default     = ""
--- a/aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl
+++ b/aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl
@ -43,6 +43,7 @@ write_files:
        --network-plugin=cni \
        --node-labels=node-role.kubernetes.io/node \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/kubernetes/kubeconfig
    permissions: '0644'
@ -69,7 +70,7 @@ runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
  - [systemctl, start, --no-block, kubelet.service]
 users:
  - default
--- a/aws/fedora-atomic/kubernetes/workers/variables.tf
+++ b/aws/fedora-atomic/kubernetes/workers/variables.tf
@ -46,6 +46,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
  type        = "string"
  default     = ""
--- a/aws/fedora-atomic/kubernetes/workers/workers.tf
+++ b/aws/fedora-atomic/kubernetes/workers/workers.tf
@ -52,6 +52,7 @@ resource "aws_launch_configuration" "worker" {
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
--- a/azure/container-linux/kubernetes/LICENSE
+++ b/azure/container-linux/kubernetes/LICENSE
@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Typhoon Authors
+Copyright (c) 2017 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -0,0 +1,22 @@
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
+
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/cl/azure/).
+
--- a/azure/container-linux/kubernetes/bootkube.tf
+++ b/azure/container-linux/kubernetes/bootkube.tf
@ -0,0 +1,13 @@
+# Self-hosted Kubernetes assets (kubeconfig, manifests)
+module "bootkube" {
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  etcd_servers          = ["${formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)}"]
+  asset_dir             = "${var.asset_dir}"
+  networking            = "flannel"
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+}
--- a/azure/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/azure/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -0,0 +1,164 @@
+---
+systemd:
+  units:
+    - name: etcd-member.service
+      enable: true
+      dropins:
+        - name: 40-etcd-cluster.conf
+          contents: |
+            [Service]
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_NAME=${etcd_name}"
+            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
+            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
+            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
+            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
+            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
+            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
+            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
+            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
+            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
+            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
+            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
+            Environment="ETCD_CLIENT_CERT_AUTH=true"
+            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
+            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
+            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
+            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
+    - name: docker.service
+      enable: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube
+        Wants=rpc-statd.service
+        [Service]
+        EnvironmentFile=/etc/kubernetes/kubelet.env
+        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --volume=resolv,kind=host,source=/etc/resolv.conf \
+          --mount volume=resolv,target=/etc/resolv.conf \
+          --volume var-lib-cni,kind=host,source=/var/lib/cni \
+          --mount volume=var-lib-cni,target=/var/lib/cni \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --insecure-options=image"
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
+        ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
+        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node-role.kubernetes.io/master \
+          --node-labels=node-role.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootkube.service
+      contents: |
+        [Unit]
+        Description=Bootstrap a Kubernetes cluster
+        ConditionPathExists=!/opt/bootkube/init_bootkube.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootkube
+        ExecStart=/opt/bootkube/bootkube-start
+        ExecStartPost=/bin/touch /opt/bootkube/init_bootkube.done
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.env
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
+          KUBELET_IMAGE_TAG=v1.12.2
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /opt/bootkube/bootkube-start
+      filesystem: root
+      mode: 0544
+      user:
+        id: 500
+      group:
+        id: 500
+      contents:
+        inline: |
+          #!/bin/bash
+          # Wrapper for bootkube start
+          set -e
+          # Move experimental manifests
+          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
+          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
+          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
+          exec /usr/bin/rkt run \
+            --trust-keys-from-https \
+            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --mount volume=assets,target=/assets \
+            --volume bootstrap,kind=host,source=/etc/kubernetes \
+            --mount volume=bootstrap,target=/etc/kubernetes \
+            $${RKT_OPTS} \
+            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            --net=host \
+            --dns=host \
+            --exec=/bootkube -- start --asset-dir=/assets "$@"
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -0,0 +1,168 @@
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "azurerm_dns_a_record" "etcds" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${var.dns_zone_group}"
+
+  # DNS Zone name where record should be created
+  zone_name = "${var.dns_zone}"
+
+  # DNS record
+  name = "${format("%s-etcd%d", var.cluster_name, count.index)}"
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  records = ["${element(azurerm_network_interface.controllers.*.private_ip_address, count.index)}"]
+}
+
+locals {
+  # Channel for a Container Linux derivative
+  # coreos-stable -> Container Linux Stable
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
+# Controller availability set to spread controllers
+resource "azurerm_availability_set" "controllers" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-controllers"
+  location                     = "${var.region}"
+  platform_fault_domain_count  = 2
+  platform_update_domain_count = 4
+  managed                      = true
+}
+
+# Controller instances
+resource "azurerm_virtual_machine" "controllers" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                = "${var.cluster_name}-controller-${count.index}"
+  location            = "${var.region}"
+  availability_set_id = "${azurerm_availability_set.controllers.id}"
+  vm_size             = "${var.controller_type}"
+
+  # boot
+  storage_image_reference {
+    publisher = "CoreOS"
+    offer     = "CoreOS"
+    sku       = "${local.channel}"
+    version   = "latest"
+  }
+
+  # storage
+  storage_os_disk {
+    name              = "${var.cluster_name}-controller-${count.index}"
+    create_option     = "FromImage"
+    caching           = "ReadWrite"
+    disk_size_gb      = "${var.disk_size}"
+    os_type           = "Linux"
+    managed_disk_type = "Premium_LRS"
+  }
+
+  # network
+  network_interface_ids = ["${element(azurerm_network_interface.controllers.*.id, count.index)}"]
+
+  os_profile {
+    computer_name  = "${var.cluster_name}-controller-${count.index}"
+    admin_username = "core"
+    custom_data    = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  }
+
+  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
+  os_profile_linux_config {
+    disable_password_authentication = true
+
+    ssh_keys {
+      path     = "/home/core/.ssh/authorized_keys"
+      key_data = "${var.ssh_authorized_key}"
+    }
+  }
+
+  # lifecycle
+  delete_os_disk_on_termination    = true
+  delete_data_disks_on_termination = true
+
+  lifecycle {
+    ignore_changes = [
+      "storage_os_disk",
+      "os_profile",
+    ]
+  }
+}
+
+# Controller NICs with public and private IPv4
+resource "azurerm_network_interface" "controllers" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                      = "${var.cluster_name}-controller-${count.index}"
+  location                  = "${azurerm_resource_group.cluster.location}"
+  network_security_group_id = "${azurerm_network_security_group.controller.id}"
+
+  ip_configuration {
+    name                          = "ip0"
+    subnet_id                     = "${azurerm_subnet.controller.id}"
+    private_ip_address_allocation = "dynamic"
+
+    # public IPv4
+    public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
+  }
+}
+
+# Add controller NICs to the controller backend address pool
+resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+  network_interface_id    = "${azurerm_network_interface.controllers.id}"
+  ip_configuration_name   = "ip0"
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+}
+
+# Controller public IPv4 addresses
+resource "azurerm_public_ip" "controllers" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-controller-${count.index}"
+  location                     = "${azurerm_resource_group.cluster.location}"
+  sku                          = "Standard"
+  public_ip_address_allocation = "static"
+}
+
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
+  count = "${var.controller_count}"
+
+  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
+
+    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  }
+}
+
+data "template_file" "etcds" {
+  count    = "${var.controller_count}"
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+  vars {
+    index        = "${count.index}"
+    cluster_name = "${var.cluster_name}"
+    dns_zone     = "${var.dns_zone}"
+  }
+}
--- a/azure/container-linux/kubernetes/lb.tf
+++ b/azure/container-linux/kubernetes/lb.tf
@ -0,0 +1,144 @@
+# DNS record for the apiserver load balancer
+resource "azurerm_dns_a_record" "apiserver" {
+  resource_group_name = "${var.dns_zone_group}"
+
+  # DNS Zone name where record should be created
+  zone_name = "${var.dns_zone}"
+
+  # DNS record
+  name = "${var.cluster_name}"
+  ttl  = 300
+
+  # IPv4 address of apiserver load balancer
+  records = ["${azurerm_public_ip.apiserver-ipv4.ip_address}"]
+}
+
+# Static IPv4 address for the apiserver frontend
+resource "azurerm_public_ip" "apiserver-ipv4" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-apiserver-ipv4"
+  location                     = "${var.region}"
+  sku                          = "Standard"
+  public_ip_address_allocation = "static"
+}
+
+# Static IPv4 address for the ingress frontend
+resource "azurerm_public_ip" "ingress-ipv4" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-ingress-ipv4"
+  location                     = "${var.region}"
+  sku                          = "Standard"
+  public_ip_address_allocation = "static"
+}
+
+# Network Load Balancer for apiservers and ingress
+resource "azurerm_lb" "cluster" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name     = "${var.cluster_name}"
+  location = "${var.region}"
+  sku      = "Standard"
+
+  frontend_ip_configuration {
+    name                 = "apiserver"
+    public_ip_address_id = "${azurerm_public_ip.apiserver-ipv4.id}"
+  }
+
+  frontend_ip_configuration {
+    name                 = "ingress"
+    public_ip_address_id = "${azurerm_public_ip.ingress-ipv4.id}"
+  }
+}
+
+resource "azurerm_lb_rule" "apiserver" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                           = "apiserver"
+  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  frontend_ip_configuration_name = "apiserver"
+
+  protocol                = "Tcp"
+  frontend_port           = 6443
+  backend_port            = 6443
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+  probe_id                = "${azurerm_lb_probe.apiserver.id}"
+}
+
+resource "azurerm_lb_rule" "ingress-http" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                           = "ingress-http"
+  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  frontend_ip_configuration_name = "ingress"
+
+  protocol                = "Tcp"
+  frontend_port           = 80
+  backend_port            = 80
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  probe_id                = "${azurerm_lb_probe.ingress.id}"
+}
+
+resource "azurerm_lb_rule" "ingress-https" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                           = "ingress-https"
+  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  frontend_ip_configuration_name = "ingress"
+
+  protocol                = "Tcp"
+  frontend_port           = 443
+  backend_port            = 443
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  probe_id                = "${azurerm_lb_probe.ingress.id}"
+}
+
+# Address pool of controllers
+resource "azurerm_lb_backend_address_pool" "controller" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "controller"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+}
+
+# Address pool of workers
+resource "azurerm_lb_backend_address_pool" "worker" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "worker"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+}
+
+# Health checks / probes
+
+# TCP health check for apiserver
+resource "azurerm_lb_probe" "apiserver" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "apiserver"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  protocol        = "Tcp"
+  port            = 6443
+
+  # unhealthy threshold
+  number_of_probes = 3
+
+  interval_in_seconds = 5
+}
+
+# HTTP health check for ingress
+resource "azurerm_lb_probe" "ingress" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "ingress"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  protocol        = "Http"
+  port            = 10254
+  request_path    = "/healthz"
+
+  # unhealthy threshold
+  number_of_probes = 3
+
+  interval_in_seconds = 5
+}
--- a/azure/container-linux/kubernetes/network.tf
+++ b/azure/container-linux/kubernetes/network.tf
@ -0,0 +1,33 @@
+# Organize cluster into a resource group
+resource "azurerm_resource_group" "cluster" {
+  name     = "${var.cluster_name}"
+  location = "${var.region}"
+}
+
+resource "azurerm_virtual_network" "network" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name          = "${var.cluster_name}"
+  location      = "${azurerm_resource_group.cluster.location}"
+  address_space = ["${var.host_cidr}"]
+}
+
+# Subnets - separate subnets for controller and workers because Azure
+# network security groups are based on IPv4 CIDR rather than instance
+# tags like GCP or security group membership like AWS
+
+resource "azurerm_subnet" "controller" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                 = "controller"
+  virtual_network_name = "${azurerm_virtual_network.network.name}"
+  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 0)}"
+}
+
+resource "azurerm_subnet" "worker" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                 = "worker"
+  virtual_network_name = "${azurerm_virtual_network.network.name}"
+  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 1)}"
+}
--- a/azure/container-linux/kubernetes/outputs.tf
+++ b/azure/container-linux/kubernetes/outputs.tf
@ -0,0 +1,32 @@
+# Outputs for Kubernetes Ingress
+
+output "ingress_static_ipv4" {
+  value       = "${azurerm_public_ip.ingress-ipv4.ip_address}"
+  description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
+# Outputs for worker pools
+
+output "region" {
+  value = "${azurerm_resource_group.cluster.location}"
+}
+
+output "resource_group_name" {
+  value = "${azurerm_resource_group.cluster.name}"
+}
+
+output "subnet_id" {
+  value = "${azurerm_subnet.worker.id}"
+}
+
+output "security_group_id" {
+  value = "${azurerm_network_security_group.worker.id}"
+}
+
+output "backend_address_pool_id" {
+  value = "${azurerm_lb_backend_address_pool.worker.id}"
+}
+
+output "kubeconfig" {
+  value = "${module.bootkube.kubeconfig}"
+}
--- a/azure/container-linux/kubernetes/require.tf
+++ b/azure/container-linux/kubernetes/require.tf
@ -0,0 +1,25 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = ">= 0.11.0"
+}
+
+provider "azurerm" {
+  version = "~> 1.17"
+}
+
+provider "local" {
+  version = "~> 1.0"
+}
+
+provider "null" {
+  version = "~> 1.0"
+}
+
+provider "template" {
+  version = "~> 1.0"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+}
--- a/azure/container-linux/kubernetes/security.tf
+++ b/azure/container-linux/kubernetes/security.tf
@ -0,0 +1,287 @@
+# Controller security group
+
+resource "azurerm_network_security_group" "controller" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name     = "${var.cluster_name}-controller"
+  location = "${azurerm_resource_group.cluster.location}"
+}
+
+resource "azurerm_network_security_rule" "controller-ssh" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-ssh"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "22"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "controller-etcd" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-etcd"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2005"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "2379-2380"
+  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow Prometheus to scrape etcd metrics
+resource "azurerm_network_security_rule" "controller-etcd-metrics" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-etcd-metrics"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2010"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "2381"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "controller-apiserver" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-apiserver"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2015"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "6443"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "controller-flannel" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-flannel"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2020"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Udp"
+  source_port_range           = "*"
+  destination_port_range      = "8472"
+  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "controller-node-exporter" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-node-exporter"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2025"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "9100"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "controller-kubelet" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-kubelet"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2030"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10250"
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-loadbalancer"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "3000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "AzureLoadBalancer"
+  destination_address_prefix  = "*"
+}
+
+resource "azurerm_network_security_rule" "controller-deny-all" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "deny-all"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "3005"
+  access                      = "Deny"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "*"
+}
+
+# Worker security group
+
+resource "azurerm_network_security_group" "worker" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name     = "${var.cluster_name}-worker"
+  location = "${azurerm_resource_group.cluster.location}"
+}
+
+resource "azurerm_network_security_rule" "worker-ssh" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-ssh"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "22"
+  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "worker-http" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-http"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2005"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "80"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "worker-https" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-https"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2010"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "443"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "worker-flannel" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-flannel"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2015"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Udp"
+  source_port_range           = "*"
+  destination_port_range      = "8472"
+  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "worker-node-exporter" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-node-exporter"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2020"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "9100"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "worker-kubelet" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-kubelet"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2025"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10250"
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-loadbalancer"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "3000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "AzureLoadBalancer"
+  destination_address_prefix  = "*"
+}
+
+resource "azurerm_network_security_rule" "worker-deny-all" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "deny-all"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "3005"
+  access                      = "Deny"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "*"
+}
--- a/azure/container-linux/kubernetes/ssh.tf
+++ b/azure/container-linux/kubernetes/ssh.tf
@ -0,0 +1,95 @@
+# Secure copy etcd TLS assets to controllers.
+resource "null_resource" "copy-controller-secrets" {
+  count = "${var.controller_count}"
+
+  depends_on = [
+    "azurerm_virtual_machine.controllers",
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = "${element(azurerm_public_ip.controllers.*.ip_address, count.index)}"
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_ca_cert}"
+    destination = "$HOME/etcd-client-ca.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_cert}"
+    destination = "$HOME/etcd-client.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_key}"
+    destination = "$HOME/etcd-client.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_cert}"
+    destination = "$HOME/etcd-server.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_key}"
+    destination = "$HOME/etcd-server.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_cert}"
+    destination = "$HOME/etcd-peer.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_key}"
+    destination = "$HOME/etcd-peer.key"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mkdir -p /etc/ssl/etcd/etcd",
+      "sudo mv etcd-client* /etc/ssl/etcd/",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
+      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
+      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
+      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
+      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
+      "sudo chown -R etcd:etcd /etc/ssl/etcd",
+      "sudo chmod -R 500 /etc/ssl/etcd",
+    ]
+  }
+}
+
+# Secure copy bootkube assets to ONE controller and start bootkube to perform
+# one-time self-hosted cluster bootstrapping.
+resource "null_resource" "bootkube-start" {
+  depends_on = [
+    "module.bootkube",
+    "module.workers",
+    "azurerm_dns_a_record.apiserver",
+    "null_resource.copy-controller-secrets",
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = "${element(azurerm_public_ip.controllers.*.ip_address, 0)}"
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    source      = "${var.asset_dir}"
+    destination = "$HOME/assets"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mv $HOME/assets /opt/bootkube",
+      "sudo systemctl start bootkube",
+    ]
+  }
+}
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -0,0 +1,117 @@
+variable "cluster_name" {
+  type        = "string"
+  description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# Azure
+
+variable "region" {
+  type        = "string"
+  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+}
+
+variable "dns_zone" {
+  type        = "string"
+  description = "Azure DNS Zone (e.g. azure.example.com)"
+}
+
+variable "dns_zone_group" {
+  type        = "string"
+  description = "Resource group where the Azure DNS Zone resides (e.g. global)"
+}
+
+# instances
+
+variable "controller_count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of controllers (i.e. masters)"
+}
+
+variable "worker_count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of workers"
+}
+
+variable "controller_type" {
+  type        = "string"
+  default     = "Standard_DS1_v2"
+  description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
+}
+
+variable "worker_type" {
+  type        = "string"
+  default     = "Standard_F1"
+  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
+}
+
+variable "os_image" {
+  type        = "string"
+  default     = "coreos-stable"
+  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
+}
+
+variable "disk_size" {
+  type        = "string"
+  default     = "40"
+  description = "Size of the disk in GB"
+}
+
+variable "worker_priority" {
+  type        = "string"
+  default     = "Regular"
+  description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
+}
+
+variable "controller_clc_snippets" {
+  type        = "list"
+  description = "Controller Container Linux Config snippets"
+  default     = []
+}
+
+variable "worker_clc_snippets" {
+  type        = "list"
+  description = "Worker Container Linux Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+  type        = "string"
+  description = "SSH public key for user 'core'"
+}
+
+variable "asset_dir" {
+  description = "Path to a directory where generated assets should be placed (contains secrets)"
+  type        = "string"
+}
+
+variable "host_cidr" {
+  description = "CIDR IPv4 range to assign to instances"
+  type        = "string"
+  default     = "10.0.0.0/16"
+}
+
+variable "pod_cidr" {
+  description = "CIDR IPv4 range to assign Kubernetes pods"
+  type        = "string"
+  default     = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+
+  type    = "string"
+  default = "10.3.0.0/16"
+}
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
--- a/azure/container-linux/kubernetes/workers.tf
+++ b/azure/container-linux/kubernetes/workers.tf
@ -0,0 +1,23 @@
+module "workers" {
+  source = "workers"
+  name   = "${var.cluster_name}"
+
+  # Azure
+  resource_group_name     = "${azurerm_resource_group.cluster.name}"
+  region                  = "${azurerm_resource_group.cluster.location}"
+  subnet_id               = "${azurerm_subnet.worker.id}"
+  security_group_id       = "${azurerm_network_security_group.worker.id}"
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+
+  count    = "${var.worker_count}"
+  vm_type  = "${var.worker_type}"
+  os_image = "${var.os_image}"
+  priority = "${var.worker_priority}"
+
+  # configuration
+  kubeconfig            = "${module.bootkube.kubeconfig}"
+  ssh_authorized_key    = "${var.ssh_authorized_key}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  clc_snippets          = "${var.worker_clc_snippets}"
+}
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -0,0 +1,122 @@
+---
+systemd:
+  units:
+    - name: docker.service
+      enable: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube
+        Wants=rpc-statd.service
+        [Service]
+        EnvironmentFile=/etc/kubernetes/kubelet.env
+        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --volume=resolv,kind=host,source=/etc/resolv.conf \
+          --mount volume=resolv,target=/etc/resolv.conf \
+          --volume var-lib-cni,kind=host,source=/var/lib/cni \
+          --mount volume=var-lib-cni,target=/var/lib/cni \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --insecure-options=image"
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
+        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node-role.kubernetes.io/node \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
+        Restart=always
+        RestartSec=5
+        [Install]
+        WantedBy=multi-user.target
+    - name: delete-node.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Waiting to delete Kubernetes node on shutdown
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/true
+        ExecStop=/etc/kubernetes/delete-node
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.env
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
+          KUBELET_IMAGE_TAG=v1.12.2
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/kubernetes/delete-node
+      filesystem: root
+      mode: 0744
+      contents:
+        inline: |
+          #!/bin/bash
+          set -e
+          exec /usr/bin/rkt run \
+            --trust-keys-from-https \
+            --volume config,kind=host,source=/etc/kubernetes \
+            --mount volume=config,target=/etc/kubernetes \
+            --insecure-options=image \
+            docker://k8s.gcr.io/hyperkube:v1.12.2 \
+            --net=host \
+            --dns=host \
+            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@ -0,0 +1,91 @@
+variable "name" {
+  type        = "string"
+  description = "Unique name for the worker pool"
+}
+
+# Azure
+
+variable "region" {
+  type        = "string"
+  description = "Must be set to the Azure Region of cluster"
+}
+
+variable "resource_group_name" {
+  type        = "string"
+  description = "Must be set to the resource group name of cluster"
+}
+
+variable "subnet_id" {
+  type        = "string"
+  description = "Must be set to the `worker_subnet_id` output by cluster"
+}
+
+variable "security_group_id" {
+  type        = "string"
+  description = "Must be set to the `worker_security_group_id` output by cluster"
+}
+
+variable "backend_address_pool_id" {
+  type        = "string"
+  description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
+}
+
+# instances
+
+variable "count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of instances"
+}
+
+variable "vm_type" {
+  type        = "string"
+  default     = "Standard_F1"
+  description = "Machine type for instances (see `az vm list-skus --location centralus`)"
+}
+
+variable "os_image" {
+  type        = "string"
+  default     = "coreos-stable"
+  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
+}
+
+variable "priority" {
+  type        = "string"
+  default     = "Regular"
+  description = "Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
+}
+
+variable "clc_snippets" {
+  type        = "list"
+  description = "Container Linux Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "kubeconfig" {
+  type        = "string"
+  description = "Must be set to `kubeconfig` output by cluster"
+}
+
+variable "ssh_authorized_key" {
+  type        = "string"
+  description = "SSH public key for user 'core'"
+}
+
+variable "service_cidr" {
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+
+  type    = "string"
+  default = "10.3.0.0/16"
+}
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -0,0 +1,113 @@
+locals {
+  # Channel for a Container Linux derivative
+  # coreos-stable -> Container Linux Stable
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
+# Workers scale set
+resource "azurerm_virtual_machine_scale_set" "workers" {
+  resource_group_name = "${var.resource_group_name}"
+
+  name                   = "${var.name}-workers"
+  location               = "${var.region}"
+  single_placement_group = false
+
+  sku {
+    name     = "${var.vm_type}"
+    tier     = "standard"
+    capacity = "${var.count}"
+  }
+
+  # boot
+  storage_profile_image_reference {
+    publisher = "CoreOS"
+    offer     = "CoreOS"
+    sku       = "${local.channel}"
+    version   = "latest"
+  }
+
+  # storage
+  storage_profile_os_disk {
+    create_option     = "FromImage"
+    caching           = "ReadWrite"
+    os_type           = "linux"
+    managed_disk_type = "Standard_LRS"
+  }
+
+  os_profile {
+    computer_name_prefix = "${var.name}-worker-"
+    admin_username       = "core"
+    custom_data          = "${data.ct_config.worker-ignition.rendered}"
+  }
+
+  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
+  os_profile_linux_config {
+    disable_password_authentication = true
+
+    ssh_keys {
+      path     = "/home/core/.ssh/authorized_keys"
+      key_data = "${var.ssh_authorized_key}"
+    }
+  }
+
+  # network
+  network_profile {
+    name                      = "nic0"
+    primary                   = true
+    network_security_group_id = "${var.security_group_id}"
+
+    ip_configuration {
+      name      = "ip0"
+      primary   = true
+      subnet_id = "${var.subnet_id}"
+
+      # backend address pool to which the NIC should be added
+      load_balancer_backend_address_pool_ids = ["${var.backend_address_pool_id}"]
+    }
+  }
+
+  # lifecycle
+  priority            = "${var.priority}"
+  upgrade_policy_mode = "Manual"
+}
+
+# Scale up or down to maintain desired number, tolerating deallocations.
+resource "azurerm_autoscale_setting" "workers" {
+  resource_group_name = "${var.resource_group_name}"
+
+  name     = "${var.name}-maintain-desired"
+  location = "${var.region}"
+
+  # autoscale
+  enabled            = true
+  target_resource_id = "${azurerm_virtual_machine_scale_set.workers.id}"
+
+  profile {
+    name = "default"
+
+    capacity {
+      minimum = "${var.count}"
+      default = "${var.count}"
+      maximum = "${var.count}"
+    }
+  }
+}
+
+# Worker Ignition configs
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.clc_snippets}"]
+}
+
+# Worker Container Linux configs
+data "template_file" "worker-config" {
+  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+
+  vars = {
+    kubeconfig            = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  }
+}
--- a/azure/ignore/.gitkeep
+++ b/azure/ignore/.gitkeep
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/bare-metal/container-linux/kubernetes/bootkube.tf
+++ b/bare-metal/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=70c28399703cb4ec8930394682400d90d733e5a5"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"

  cluster_name                    = "${var.cluster_name}"
  api_servers                     = ["${var.k8s_domain_name}"]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@ -97,6 +97,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -123,7 +124,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.2
+          KUBELET_IMAGE_TAG=v1.12.2
    - path: /etc/hostname
      filesystem: root
      mode: 0644
@ -163,8 +164,6 @@ storage:
            --net=host \
            --dns=host \
            --exec=/bootkube -- start --asset-dir=/assets "$@"
-networkd:
-  ${networkd_content}
 passwd:
  users:
    - name: core
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -69,6 +69,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -84,7 +85,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.2
+          KUBELET_IMAGE_TAG=v1.12.2
    - path: /etc/hostname
      filesystem: root
      mode: 0644
@ -96,8 +97,6 @@ storage:
      contents:
        inline: |
          fs.inotify.max_user_watches=16184
-networkd:
-  ${networkd_content}
 passwd:
  users:
    - name: core
--- a/bare-metal/container-linux/kubernetes/groups.tf
+++ b/bare-metal/container-linux/kubernetes/groups.tf
@ -3,7 +3,7 @@ resource "matchbox_group" "install" {

  name = "${format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"

-  profile = "${local.flavor == "flatcar" ? element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
+  profile = "${local.flavor == "flatcar" ? var.cached_install == "true" ? element(matchbox_profile.cached-flatcar-linux-install.*.name, count.index) : element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"

  selector {
    mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -49,7 +49,7 @@ data "template_file" "container-linux-install-configs" {
 }

 // Container Linux Install profile (from matchbox /assets cache)
-// Note: Admin must have downloaded os_version into matchbox assets.
+// Note: Admin must have downloaded os_version into matchbox assets/coreos.
 resource "matchbox_profile" "cached-container-linux-install" {
  count = "${length(var.controller_names) + length(var.worker_names)}"
  name  = "${format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
@ -87,7 +87,7 @@ data "template_file" "cached-container-linux-install-configs" {
    ssh_authorized_key  = "${var.ssh_authorized_key}"

    # profile uses -b baseurl to install from matchbox cache
-    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/coreos"
+    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/${local.flavor}"
  }
 }

@ -114,22 +114,46 @@ resource "matchbox_profile" "flatcar-install" {
  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
 }

+// Flatcar Linux Install profile (from matchbox /assets cache)
+// Note: Admin must have downloaded os_version into matchbox assets/flatcar.
+resource "matchbox_profile" "cached-flatcar-linux-install" {
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+  name  = "${format("%s-cached-flatcar-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+
+  kernel = "/assets/flatcar/${var.os_version}/flatcar_production_pxe.vmlinuz"
+
+  initrd = [
+    "/assets/flatcar/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
+  ]
+
+  args = [
+    "initrd=flatcar_production_pxe_image.cpio.gz",
+    "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
+    "flatcar.first_boot=yes",
+    "console=tty0",
+    "console=ttyS0",
+    "${var.kernel_args}",
+  ]
+
+  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
+}
+
 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
-  count                  = "${length(var.controller_names)}"
-  name                   = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
+  count        = "${length(var.controller_names)}"
+  name         = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
  raw_ignition = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
 }

 data "ct_config" "controller-ignitions" {
-  count = "${length(var.controller_names)}"
-  content = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  count        = "${length(var.controller_names)}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
  pretty_print = false
+
  # Must use direct lookup. Cannot use lookup(map, key) since it only works for flat maps
  snippets = ["${local.clc_map[element(var.controller_names, count.index)]}"]
 }

-
 data "template_file" "controller-configs" {
  count = "${length(var.controller_names)}"

@ -142,24 +166,21 @@ data "template_file" "controller-configs" {
    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
    cluster_domain_suffix = "${var.cluster_domain_suffix}"
    ssh_authorized_key    = "${var.ssh_authorized_key}"
-
-    # Terraform evaluates both sides regardless and element cannot be used on 0 length lists
-    networkd_content = "${length(var.controller_networkds) == 0 ? "" : element(concat(var.controller_networkds, list("")), count.index)}"
  }
 }

 // Kubernetes Worker profiles
 resource "matchbox_profile" "workers" {
-  count                  = "${length(var.worker_names)}"
-  name                   = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
+  count        = "${length(var.worker_names)}"
+  name         = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
  raw_ignition = "${element(data.ct_config.worker-ignitions.*.rendered, count.index)}"
 }

-
 data "ct_config" "worker-ignitions" {
-  count = "${length(var.worker_names)}"
-  content = "${element(data.template_file.worker-configs.*.rendered, count.index)}"
+  count        = "${length(var.worker_names)}"
+  content      = "${element(data.template_file.worker-configs.*.rendered, count.index)}"
  pretty_print = false
+
  # Must use direct lookup. Cannot use lookup(map, key) since it only works for flat maps
  snippets = ["${local.clc_map[element(var.worker_names, count.index)]}"]
 }
@ -174,9 +195,6 @@ data "template_file" "worker-configs" {
    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
    cluster_domain_suffix = "${var.cluster_domain_suffix}"
    ssh_authorized_key    = "${var.ssh_authorized_key}"
-
-    # Terraform evaluates both sides regardless and element cannot be used on 0 length lists
-    networkd_content = "${length(var.worker_networkds) == 0 ? "" : element(concat(var.worker_networkds, list("")), count.index)}"
  }
 }

@ -185,12 +203,13 @@ locals {
  # Default Container Linux config snippets map every node names to list("\n") so
  # all lookups succeed
  clc_defaults = "${zipmap(concat(var.controller_names, var.worker_names), chunklist(data.template_file.clc-default-snippets.*.rendered, 1))}"
+
  # Union of the default and user specific snippets, later overrides prior.
  clc_map = "${merge(local.clc_defaults, var.clc_snippets)}"
 }

 // Horrible hack to generate a Terraform list of node count length
 data "template_file" "clc-default-snippets" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count    = "${length(var.controller_names) + length(var.worker_names)}"
  template = "\n"
 }
--- a/bare-metal/container-linux/kubernetes/variables.tf
+++ b/bare-metal/container-linux/kubernetes/variables.tf
@ -24,39 +24,39 @@ variable "os_version" {
 # Terraform's crude "type system" does not properly support lists of maps so we do this.

 variable "controller_names" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of controller names (e.g. [node1])"
 }

 variable "controller_macs" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of controller identifying MAC addresses (e.g. [52:54:00:a1:9c:ae])"
 }

 variable "controller_domains" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of controller FQDNs (e.g. [node1.example.com])"
 }

 variable "worker_names" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of worker names (e.g. [node2, node3])"
 }

 variable "worker_macs" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of worker identifying MAC addresses (e.g. [52:54:00:b2:2f:86, 52:54:00:c3:61:77])"
 }

 variable "worker_domains" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of worker FQDNs (e.g. [node2.example.com, node3.example.com])"
 }

 variable "clc_snippets" {
-  type = "map"
+  type        = "map"
  description = "Map from machine names to lists of Container Linux Config snippets"
-  default = {}
+  default     = {}
 }

 # configuration
@ -141,17 +141,3 @@ variable "kernel_args" {
  type        = "list"
  default     = []
 }
-
-# unofficial, undocumented, unsupported, temporary
-
-variable "controller_networkds" {
-  type        = "list"
-  description = "Controller Container Linux config networkd section"
-  default     = []
-}
-
-variable "worker_networkds" {
-  type        = "list"
-  description = "Worker Container Linux config networkd section"
-  default     = []
-}
--- a/bare-metal/fedora-atomic/kubernetes/README.md
+++ b/bare-metal/fedora-atomic/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/bare-metal/fedora-atomic/kubernetes/bootkube.tf
+++ b/bare-metal/fedora-atomic/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=70c28399703cb4ec8930394682400d90d733e5a5"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${var.k8s_domain_name}"]
--- a/bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -51,6 +51,7 @@ write_files:
        --node-labels=node-role.kubernetes.io/master \
        --node-labels=node-role.kubernetes.io/controller="true" \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
@ -83,8 +84,8 @@ runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
  - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
  - [systemctl, start, --no-block, etcd.service]
  - [systemctl, enable, kubelet.path]
--- a/bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
+++ b/bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
@ -29,6 +29,7 @@ write_files:
        --network-plugin=cni \
        --node-labels=node-role.kubernetes.io/node \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
    content: |
@ -59,7 +60,7 @@ runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
  - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
  - [systemctl, enable, kubelet.path]
  - [systemctl, start, --no-block, kubelet.path]
 users:
--- a/bare-metal/fedora-atomic/kubernetes/variables.tf
+++ b/bare-metal/fedora-atomic/kubernetes/variables.tf
@ -25,32 +25,32 @@ EOD
 # Terraform's crude "type system" does not properly support lists of maps so we do this.

 variable "controller_names" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of controller names (e.g. [node1])"
 }

 variable "controller_macs" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of controller identifying MAC addresses (e.g. [52:54:00:a1:9c:ae])"
 }

 variable "controller_domains" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of controller FQDNs (e.g. [node1.example.com])"
 }

 variable "worker_names" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of worker names (e.g. [node2, node3])"
 }

 variable "worker_macs" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of worker identifying MAC addresses (e.g. [52:54:00:b2:2f:86, 52:54:00:c3:61:77])"
 }

 variable "worker_domains" {
-  type = "list"
+  type        = "list"
  description = "Ordered list of worker FQDNs (e.g. [node2.example.com, node3.example.com])"
 }

--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/digital-ocean/container-linux/kubernetes/bootkube.tf
+++ b/digital-ocean/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=70c28399703cb4ec8930394682400d90d733e5a5"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_IMAGE_TAG=v3.3.10"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -100,6 +100,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -128,7 +129,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.2
+          KUBELET_IMAGE_TAG=v1.12.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -72,6 +72,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -98,7 +99,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.11.2
+          KUBELET_IMAGE_TAG=v1.12.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -116,7 +117,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.11.2 \
+            docker://k8s.gcr.io/hyperkube:v1.12.2 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/controllers.tf
+++ b/digital-ocean/container-linux/kubernetes/controllers.tf
@ -44,12 +44,18 @@ resource "digitalocean_droplet" "controllers" {
  ipv6               = true
  private_networking = true

-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
  ssh_keys  = ["${var.ssh_fingerprints}"]

  tags = [
    "${digitalocean_tag.controllers.id}",
  ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }

 # Tag to label controllers
@ -57,8 +63,16 @@ resource "digitalocean_tag" "controllers" {
  name = "${var.cluster_name}-controller"
 }

-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
  count = "${var.controller_count}"

  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@ -85,11 +99,3 @@ data "template_file" "etcds" {
    dns_zone     = "${var.dns_zone}"
  }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-
-  snippets = ["${var.controller_clc_snippets}"]
-}
--- a/digital-ocean/container-linux/kubernetes/outputs.tf
+++ b/digital-ocean/container-linux/kubernetes/outputs.tf
@ -3,7 +3,8 @@ output "controllers_dns" {
 }

 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }

 output "controllers_ipv4" {
--- a/digital-ocean/container-linux/kubernetes/require.tf
+++ b/digital-ocean/container-linux/kubernetes/require.tf
@ -5,7 +5,7 @@ terraform {
 }

 provider "digitalocean" {
-  version = "~> 0.1.2"
+  version = "~> 1.0"
 }

 provider "local" {
--- a/digital-ocean/container-linux/kubernetes/workers.tf
+++ b/digital-ocean/container-linux/kubernetes/workers.tf
@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
  count = "${var.worker_count}"

  # DNS zone where record should be created
@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
  value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }

+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
  count = "${var.worker_count}"
@ -25,12 +37,16 @@ resource "digitalocean_droplet" "workers" {
  ipv6               = true
  private_networking = true

-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"
  ssh_keys  = ["${var.ssh_fingerprints}"]

  tags = [
    "${digitalocean_tag.workers.id}",
  ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }

 # Tag to label workers
@ -38,8 +54,15 @@ resource "digitalocean_tag" "workers" {
  name = "${var.cluster_name}-worker"
 }

-# Worker Container Linux Config
-data "template_file" "worker_config" {
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.worker_clc_snippets}"]
+}
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars = {
@ -47,9 +70,3 @@ data "template_file" "worker_config" {
    cluster_domain_suffix = "${var.cluster_domain_suffix}"
  }
 }
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
-  pretty_print = false
-  snippets     = ["${var.worker_clc_snippets}"]
-}
--- a/digital-ocean/fedora-atomic/kubernetes/README.md
+++ b/digital-ocean/fedora-atomic/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.11.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/digital-ocean/fedora-atomic/kubernetes/bootkube.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=70c28399703cb4ec8930394682400d90d733e5a5"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -65,6 +65,7 @@ write_files:
        --node-labels=node-role.kubernetes.io/master \
        --node-labels=node-role.kubernetes.io/controller="true" \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
@ -89,8 +90,8 @@ bootcmd:
  - [modprobe, ip_vs]
 runcmd:
  - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
  - [systemctl, start, --no-block, etcd.service]
  - [systemctl, enable, cloud-metadata.service]
--- a/digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
+++ b/digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
@ -43,6 +43,7 @@ write_files:
        --network-plugin=cni \
        --node-labels=node-role.kubernetes.io/node \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
    content: |
@ -66,7 +67,7 @@ bootcmd:
 runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
  - [systemctl, enable, kubelet.path]
  - [systemctl, start, --no-block, kubelet.path]
 users:
--- a/digital-ocean/fedora-atomic/kubernetes/controllers.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/controllers.tf
@ -50,6 +50,12 @@ resource "digitalocean_droplet" "controllers" {
  tags = [
    "${digitalocean_tag.controllers.id}",
  ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }

 # Tag to label controllers
--- a/digital-ocean/fedora-atomic/kubernetes/outputs.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/outputs.tf
@ -3,7 +3,8 @@ output "controllers_dns" {
 }

 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }

 output "controllers_ipv4" {
--- a/digital-ocean/fedora-atomic/kubernetes/require.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/require.tf
@ -5,7 +5,7 @@ terraform {
 }

 provider "digitalocean" {
-  version = "~> 0.1.2"
+  version = "~> 1.0"
 }

 provider "local" {
--- a/digital-ocean/fedora-atomic/kubernetes/workers.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/workers.tf
@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
  count = "${var.worker_count}"

  # DNS zone where record should be created
@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
  value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }

+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
  count = "${var.worker_count}"
@ -31,6 +43,10 @@ resource "digitalocean_droplet" "workers" {
  tags = [
    "${digitalocean_tag.workers.id}",
  ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }

 # Tag to label workers
--- a/docs/addons/ingress.md
+++ b/docs/addons/ingress.md
@ -4,7 +4,7 @@ Nginx Ingress controller pods accept and demultiplex HTTP, HTTPS, TCP, or UDP tr

 ## AWS

-On AWS, a network load balancer (NLB) distributes traffic across a target group  of worker nodes running an Ingress controller deployment on host ports 80 and 443. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a health Ingress controller receive traffic.
+On AWS, a network load balancer (NLB) distributes TCP traffic across two target groups (port 80 and 443) of worker nodes running an Ingress controller deployment. Security groups rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.

 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.

@ -12,15 +12,15 @@ Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, de
 kubectl apply -R -f addons/nginx-ingress/aws
 ```

-For each application, add a DNS CNAME resolving to the ELB's DNS record.
+For each application, add a DNS CNAME resolving to the NLB's DNS record.

 ```
 app1.example.com -> tempest-ingress.123456.us-west2.elb.amazonaws.com
-aap2.example.com -> tempest-ingress.123456.us-west2.elb.amazonaws.com
+app2.example.com -> tempest-ingress.123456.us-west2.elb.amazonaws.com
 app3.example.com -> tempest-ingress.123456.us-west2.elb.amazonaws.com
 ```

-Find the ELB's DNS name through the console or use the Typhoon module's output `ingress_dns_name`. For example, you might use Terraform to manage a Google Cloud DNS record:
+Find the NLB's DNS name through the console or use the Typhoon module's output `ingress_dns_name`. For example, you might use Terraform to manage a Google Cloud DNS record:

 ```tf
 resource "google_dns_record_set" "some-application" {
@ -35,52 +35,25 @@ resource "google_dns_record_set" "some-application" {
 }
 ```

-## Digital Ocean
+## Azure

-On Digital Ocean, a DNS A record (e.g. `nemo-workers.example.com`) resolves to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
-
-Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, default backend, and namespace.
-
-```
-kubectl apply -R -f addons/nginx-ingress/digital-ocean
-```
-
-For each application, add a CNAME record resolving to the worker(s) DNS record. Use the Typhoon module's output `workers_dns` to find the worker DNS value. For example, you might use Terraform to manage a Google Cloud DNS record:
-
-```tf
-resource "google_dns_record_set" "some-application" {
-  # DNS zone name
-  managed_zone = "example-zone"
-
-  # DNS record
-  name    = "app.example.com."
-  type    = "CNAME"
-  ttl     = 300
-  rrdatas = ["${module.digital-ocean-nemo.workers_dns}."]
-}
-```
-
-[^1]: Digital Ocean does offers load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
-
-## Google Cloud
-
-On Google Cloud, a network load balancer distributes traffic across worker nodes (i.e. a target pool of backends) running an Ingress controller deployment on host ports 80 and 443. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure the target pool only includes worker nodes with a healthy Nginx Ingress controller.
+On Azure, a load balancer distributes traffic across a backend address pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.

 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.

 ```
-kubectl apply -R -f addons/nginx-ingress/google-cloud
+kubectl apply -R -f addons/nginx-ingress/azure
 ```

-For each application, add a DNS record resolving to the network load balancer's IPv4 address.
+For each application, add a DNS record resolving to the load balancer's IPv4 address.

 ```
 app1.example.com -> 11.22.33.44
-aap2.example.com -> 11.22.33.44
+app2.example.com -> 11.22.33.44
 app3.example.com -> 11.22.33.44
 ```

-Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's output `ingress_static_ipv4`. For example, you might use Terraform to manage a Google Cloud DNS record:
+Find the load balancer's IPv4 address with the Azure console or use the Typhoon module's output `ingress_static_ipv4`. For example, you might use Terraform to manage a Google Cloud DNS record:

 ```tf
 resource "google_dns_record_set" "some-application" {
@ -91,7 +64,7 @@ resource "google_dns_record_set" "some-application" {
  name    = "app.example.com."
  type    = "A"
  ttl     = 300
-  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv4}"]
+  rrdatas = ["${module.azure-ramius.ingress_static_ipv4}"]
 }
 ```

@ -125,3 +98,77 @@ resource "google_dns_record_set" "some-application" {
  rrdatas = ["SOME-WAN-IP"]
 }
 ```
+
+## Digital Ocean
+
+On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
+
+Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, default backend, and namespace.
+
+```
+kubectl apply -R -f addons/nginx-ingress/digital-ocean
+```
+
+For each application, add a CNAME record resolving to the worker(s) DNS record. Use the Typhoon module's output `workers_dns` to find the worker DNS value. For example, you might use Terraform to manage a Google Cloud DNS record:
+
+```tf
+resource "google_dns_record_set" "some-application" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "CNAME"
+  ttl     = 300
+  rrdatas = ["${module.digital-ocean-nemo.workers_dns}."]
+}
+```
+
+!!! note
+    Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
+
+[^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
+
+## Google Cloud
+
+On Google Cloud, a TCP Proxy load balancer distributes IPv4 and IPv6 TCP traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.
+
+Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
+
+```
+kubectl apply -R -f addons/nginx-ingress/google-cloud
+```
+
+For each application, add DNS A records resolving to the load balancer's IPv4 address and DNS AAAA records resolving to the load balancer's IPv6 address.
+
+```
+app1.example.com -> 11.22.33.44
+app2.example.com -> 11.22.33.44
+app3.example.com -> 11.22.33.44
+```
+
+Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's outputs `ingress_static_ipv4` and `ingress_static_ipv6`. For example, you might use Terraform to manage a Google Cloud DNS record:
+
+```tf
+resource "google_dns_record_set" "app-record-a" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "A"
+  ttl     = 300
+  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv4}"]
+}
+
+resource "google_dns_record_set" "app-record-aaaa" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "AAAA"
+  ttl     = 300
+  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv6}"]
+}
+```
--- a/docs/addons/prometheus.md
+++ b/docs/addons/prometheus.md
@ -47,4 +47,4 @@ Visit [127.0.0.1:9090](http://127.0.0.1:9090) to query [expressions](http://127.
 <br/>
 ![Prometheus Alerts](../img/prometheus-alerts.png)

-Use [Grafana](/addons/grafana.md) to view or build dashboards that use Prometheus as the datasource.
+Use [Grafana](/addons/grafana/) to view or build dashboards that use Prometheus as the datasource.
--- a/docs/advanced/customization.md
+++ b/docs/advanced/customization.md
@ -19,7 +19,7 @@ Clusters are kept to a minimal Kubernetes control plane by offering components l

 Container Linux Configs (CLCs) declare how a Container Linux instance's disk should be provisioned on first boot from disk. CLCs define disk partitions, filesystems, files, systemd units, dropins, networkd configs, mount units, raid arrays, and users. Typhoon creates controller and worker instances with base Container Linux Configs to create a minimal, secure Kubernetes cluster on each platform.

-Typhoon AWS, Google Cloud, and Digital Ocean give users the ability to provide CLC *snippets* - valid Container Linux Configs that are validated and additively merged into the Typhoon base config during `terraform plan`. This allows advanced host customizations and experimentation.
+Typhoon AWS, Azure, bare-metal, DigitalOcean, and Google Cloud support CLC *snippets* - valid Container Linux Configs that are validated and additively merged into the Typhoon base config during `terraform plan`. This allows advanced host customizations and experimentation.

 #### Examples

@ -69,7 +69,7 @@ View the Container Linux Config [format](https://coreos.com/os/docs/1576.4.0/con

 Write Container Linux Configs *snippets* as files in the repository where you keep Terraform configs for clusters (perhaps in a `clc` or `snippets` subdirectory). You may organize snippets in multiple files as desired, provided they are each valid.

-[AWS](/cl/aws/#cluster), [Google Cloud](/cl/google-cloud/#cluster), and [Digital Ocean](/cl/digital-ocean/#cluster) clusters allow populating a list of `controller_clc_snippets` or `worker_clc_snippets`.
+[AWS](/cl/aws/#cluster), [Azure](/cl/azure/#cluster), [DigitalOcean](/cl/digital-ocean/#cluster), and [Google Cloud](/cl/google-cloud/#cluster) clusters allow populating a list of `controller_clc_snippets` or `worker_clc_snippets`.

 ```
 module "digital-ocean-nemo" {
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -1,11 +1,12 @@
 # Worker Pools

-Typhoon AWS and Google Cloud allow additional groups of workers to be defined and joined to a cluster. For example, add worker pools of instances with different types, disk sizes, Container Linux channels, or preemptibility modes.
+Typhoon AWS, Azure, and Google Cloud allow additional groups of workers to be defined and joined to a cluster. For example, add worker pools of instances with different types, disk sizes, Container Linux channels, or preemptibility modes.

 Internal Terraform Modules:

 * `aws/container-linux/kubernetes/workers`
 * `aws/fedora-atomic/kubernetes/workers`
+* `azure/container-linux/kubernetes/workers`
 * `google-cloud/container-linux/kubernetes/workers`
 * `google-cloud/fedora-atomic/kubernetes/workers`

@ -15,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo

 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.11.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.12.2"
  
  providers = {
    aws = "aws.default"
@ -31,6 +32,7 @@ module "tempest-worker-pool" {
  kubeconfig         = "${module.aws-tempest.kubeconfig}"
  ssh_authorized_key = "${var.ssh_authorized_key}"

+  # optional
  count         = 2
  instance_type = "m5.large"
  os_image      = "coreos-beta"
@ -43,7 +45,7 @@ Apply the change.
 terraform apply
 ```

-Verify an auto-scaling group of workers join the cluster within a few minutes.
+Verify an auto-scaling group of workers joins the cluster within a few minutes.

 ### Variables

@ -53,10 +55,10 @@ The AWS internal `workers` module supports a number of [variables](https://githu

 | Name | Description | Example |
 |:-----|:------------|:--------|
+| name | Unique name (distinct from cluster name) | "tempest-m5s" |
 | vpc_id | Must be set to `vpc_id` output by cluster | "${module.cluster.vpc_id}" |
 | subnet_ids | Must be set to `subnet_ids` output by cluster | "${module.cluster.subnet_ids}" |
 | security_groups | Must be set to `worker_security_groups` output by cluster | "${module.cluster.worker_security_groups}" |
-| name | Unique name (distinct from cluster name) | "tempest-m5s" |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | "${module.cluster.kubeconfig}" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

@ -74,20 +76,90 @@ The AWS internal `workers` module supports a number of [variables](https://githu

 Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/) or per-region and per-type [spot prices](https://aws.amazon.com/ec2/spot/pricing/).

+## Azure
+
+Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define a worker pool using the Azure internal `workers` module.
+
+```tf
+module "ramius-worker-pool" {
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.12.2"
+  
+  providers = {
+    azurerm = "azurerm.default"
+  }
+
+  # Azure
+  region                  = "${module.azure-ramius.region}"
+  resource_group_name     = "${module.azure-ramius.resource_group_name}"
+  subnet_id               = "${module.azure-ramius.subnet_id}"
+  security_group_id       = "${module.azure-ramius.security_group_id}"
+  backend_address_pool_id = "${module.azure-ramius.backend_address_pool_id}"
+
+  # configuration
+  name               = "ramius-low-priority"
+  kubeconfig         = "${module.azure-ramius.kubeconfig}"
+  ssh_authorized_key = "${var.ssh_authorized_key}"
+
+  # optional
+  count    = 2
+  vm_type  = "Standard_F4"
+  priority = "Low"
+}
+```
+
+Apply the change.
+
+```
+terraform apply
+```
+
+Verify a scale set of workers joins the cluster within a few minutes.
+
+### Variables
+
+The Azure internal `workers` module supports a number of [variables](https://github.com/poseidon/typhoon/blob/master/azure/container-linux/kubernetes/workers/variables.tf).
+
+#### Required
+
+| Name | Description | Example |
+|:-----|:------------|:--------|
+| name | Unique name (distinct from cluster name) | "ramius-f4" |
+| region | Must be set to `region` output by cluster | "${module.cluster.region}" |
+| resource_group_name | Must be set to `resource_group_name` output by cluster | "${module.cluster.resource_group_name}" |
+| subnet_id | Must be set to `subnet_id` output by cluster | "${module.cluster.subnet_id}" |
+| security_group_id | Must be set to `security_group_id` output by cluster | "${module.cluster.security_group_id}" |
+| backend_address_pool_id | Must be set to `backend_address_pool_id` output by cluster | "${module.cluster.backend_address_pool_id}" |
+| kubeconfig | Must be set to `kubeconfig` output by cluster | "${module.cluster.kubeconfig}" |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+
+#### Optional
+
+| Name | Description | Default | Example |
+|:-----|:------------|:--------|:--------|
+| count | Number of instances | 1 | 3 |
+| vm_type | Machine type for instances | "Standard_F1" | See below |
+| os_image | Channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha |
+| priority | Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Low |
+| clc_snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by coredns. | "cluster.local" | "k8s.example.com" |
+
+Check the list of valid [machine types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) and their [specs](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general). Use `az vm list-skus` to get the identifier.
+
 ## Google Cloud

 Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#cluster). Define a worker pool using the Google Cloud internal `workers` module.

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.11.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.12.2"

  providers = {
    google = "google.default"
  }

  # Google Cloud
-  region       = "us-central1"
+  region       = "europe-west2"
  network      = "${module.google-cloud-yavin.network_name}"
  cluster_name = "yavin"

@ -96,6 +168,7 @@ module "yavin-worker-pool" {
  kubeconfig         = "${module.google-cloud-yavin.kubeconfig}"
  ssh_authorized_key = "${var.ssh_authorized_key}"
  
+  # optional
  count        = 2
  machine_type = "n1-standard-16"
  os_image     = "coreos-beta"
@ -114,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.11.2
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.11.2
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.11.2
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.11.2
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.11.2
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.12.2
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.12.2
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.12.2
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.12.2
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.12.2
 ```

 ### Variables
@ -129,13 +202,15 @@ The Google Cloud internal `workers` module supports a number of [variables](http

 | Name | Description | Example |
 |:-----|:------------|:--------|
-| region | Must be set to `region` of cluster | "us-central1" |
-| network | Must be set to `network_name` output by cluster | "${module.cluster.network_name}" |
 | name | Unique name (distinct from cluster name) | "yavin-16x" |
+| region | Region for the worker pool instances. May differ from the cluster's region | "europe-west2" |
+| network | Must be set to `network_name` output by cluster | "${module.cluster.network_name}" |
 | cluster_name | Must be set to `cluster_name` of cluster | "yavin" |
 | kubeconfig | Must be set to `kubeconfig` output by cluster | "${module.cluster.kubeconfig}" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

+Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`.
+
 #### Optional

 | Name | Description | Default | Example |
--- a/docs/architecture/aws.md
+++ b/docs/architecture/aws.md
@ -0,0 +1,13 @@
+# AWS
+
+## IPv6
+
+Status of IPv6 on Typhoon AWS clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | No        |
+
+* AWS Network Load Balancers do not support `dualstack`.
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Dalton Hubble	78c9fdc18f	Update mkdocs-material docs theme version	2018-10-28 19:45:58 -07:00
Dalton Hubble	884c8b39dc	Update Grafana from v5.3.1 to v5.3.2 * https://github.com/grafana/grafana/releases/tag/v5.3.2	2018-10-28 19:44:22 -07:00
Dalton Hubble	0e71f7e565	Ignore controller user_data changes to allow plugin updates * Updating the `terraform-provider-ct` plugin is known to produce a `user_data` diff in all pre-existing clusters. Applying the diff to pre-existing cluster destroys controller nodes * Ignore changes to controller `user_data`. Once all managed clusters use a release containing this change, it is possible to update the `terraform-provider-ct` plugin (worker `user_data` will still be modified) * Changing the module `ref` for an existing cluster and re-applying is still NOT supported (although this PR would protect controllers from being destroyed)	2018-10-28 16:48:12 -07:00
Dalton Hubble	8c4200d425	Add DigitalOcean AAAA DNS records on Fedora Atomic	2018-10-28 14:57:31 -07:00
Dalton Hubble	5be5b261e2	Add an IPv6 address and forwarding rules on Google Cloud * Allowing serving IPv6 applications via Kubernetes Ingress on Typhoon Google Cloud clusters * Add `ingress_static_ipv6` output variable for use in AAAA DNS records	2018-10-28 14:30:58 -07:00
Dalton Hubble	f034ef90ae	Add DigitalOcean AAAA DNS records resolving to workers * Improve the workers "round-robin" DNS FQDN that is created with each cluster by adding AAAA records * CNAME's resolving to the DigitalOcean `workers_dns` output can be followed to find a droplet's IPv4 or IPv6 address * The CNI portmap plugin doesn't support IPv6. Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon with `hostNetwork: true`	2018-10-27 23:09:24 -07:00
Dalton Hubble	3bba1ba0dc	Use new azurerm_network_interface_backend_address_pool_association * Require terraform-provider-azurerm v1.17+ * Inline load_balancer_backend_address_pools_ids is deprecated and scheduled for removal in the v2.0 provider * https://github.com/terraform-providers/terraform-provider-azurerm/pull/2079	2018-10-27 22:55:05 -07:00
Dalton Hubble	dbe7604b67	Add primary field to ip_configuration required by Azure * Required by terraform-provider-azurerm v1.17+ * https://github.com/terraform-providers/terraform-provider-azurerm/pull/2035	2018-10-27 16:44:44 -07:00
Dalton Hubble	9b405a19b2	Fix minor naming inconsistencies in Ignition and CLC data	2018-10-27 16:24:59 -07:00
Tamer Fahmy	bfa1a679eb	Name AWS and DigitalOcean Ignition data sources consistently	2018-10-27 16:14:44 -07:00
Dalton Hubble	f1da0731d8	Update Kubernetes from v1.12.1 to v1.12.2 * Update CoreDNS from v1.2.2 to v1.2.4 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122 * https://coredns.io/2018/10/17/coredns-1.2.4-release/ * https://coredns.io/2018/10/16/coredns-1.2.3-release/	2018-10-27 15:47:57 -07:00
Dalton Hubble	d641a058fe	Update Calico from v3.2.3 to v3.3.0 * https://docs.projectcalico.org/v3.3/releases/	2018-10-23 20:30:30 -07:00
Dalton Hubble	99a6d5478b	Disable Kubelet read-only port 10255 * We can finally disable the Kubelet read-only port 10255! * Journey: https://github.com/poseidon/typhoon/issues/322#issuecomment-431073073	2018-10-18 21:14:14 -07:00
Dalton Hubble	bc750aec33	Configure Heapster to source metrics from Kubelet authenticated API * Heapster can now get nodes (i.e. kubelets) from the apiserver and source metrics from the Kubelet authenticated API (10250) instead of the Kubelet HTTP read-only API (10255) * https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md * Use the heapster service account token via Kubelet bearer token authn/authz. * Permit Heapster to skip CA verification. The CA cert does not contain IP SANs and cannot since nodes get random IPs that aren't known upfront. Heapster obtains the node list from the apiserver, so the risk of spoofing a node is limited. For the same reason, Prometheus scrapes must skip CA verification for scraping Kubelet's provided by the apiserver. * https://github.com/poseidon/typhoon/blob/v1.12.1/addons/prometheus/config.yaml#L68 * Create a heapster ClusterRole to work around the default Kubernetes `system:heapster` ClusterRole lacking the proper GET `nodes/stats` access. See https://github.com/kubernetes/heapster/issues/1936	2018-10-18 21:03:01 -07:00
Dalton Hubble	d55bfd5589	Fix CoreDNS AntiAffinity spec to prefer spreading replicas * Pods were still being scheduled at random due to a typo	2018-10-17 22:19:57 -07:00
Robert Fairburn	0be4673e44	Add disk_iops variable for AWS * Setting disk_iops is required for disk_type io1 * https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#EBSVolumeTypes	2018-10-17 22:18:54 -07:00
Dalton Hubble	3b44972d78	Add links to header to CHANGES	2018-10-17 09:08:58 -07:00
Dalton Hubble	0127ee82c1	Update nginx-ingress from v0.19.0 to v0.20.0	2018-10-16 21:35:29 -07:00
Dalton Hubble	a10d6977b8	Update Prometheus from v2.4.2 to v2.4.3 * https://github.com/prometheus/prometheus/releases/tag/v2.4.3	2018-10-16 21:29:41 -07:00
Dalton Hubble	05fe923c14	Update Grafana from v5.3.0 to v5.3.1 * https://github.com/grafana/grafana/releases/tag/v5.3.1	2018-10-16 21:23:44 -07:00
Michael Schubert	d10620fb58	Add support for Flatcar Linux bare-metal cached_install * Support bare-metal cached_install=true mode with Flatcar Linux where assets are fetched from the Matchbox assets cache instead of from the upstream Flatcar download server * Skipped in original Flatcar support to keep it simple https://github.com/poseidon/typhoon/pull/209	2018-10-16 21:15:24 -07:00
Dalton Hubble	9b6113a058	Update Kubernetes from v1.11.3 to v1.12.1 * Mount an empty dir for the controller-manager to work around https://github.com/kubernetes/kubernetes/issues/68973 * Update coreos/pod-checkpointer to strip affinity from checkpointed pod manifests. Kubernetes v1.12.0-rc.1 introduced a default affinity that appears on checkpointed manifests; but it prevented scheduling and checkpointed pods should not have an affinity, they're run directly by the Kubelet on the local node * https://github.com/kubernetes-incubator/bootkube/issues/1001 * https://github.com/kubernetes/kubernetes/pull/68173	2018-10-16 20:28:13 -07:00
Dalton Hubble	5eb4078d68	Add docker/default seccomp to control plane and addons * Annotate pods, deployments, and daemonsets to start containers with the Docker runtime's default seccomp profile * Overrides Kubernetes default behavior which started containers with seccomp=unconfined * https://docs.docker.com/engine/security/seccomp/#pass-a-profile-for-a-container	2018-10-16 20:07:29 -07:00
Dalton Hubble	8f0d2b5db4	Update Grafana from v5.2.4 to v5.3.0	2018-10-13 23:03:31 -07:00
Dalton Hubble	2e89e161e9	Remove Azure admin_password (disabled) now that its optional * Requires terraform-provider-azurerm v1.16.0 or higher https://github.com/terraform-providers/terraform-provider-azurerm/pull/1958	2018-10-13 22:40:58 -07:00
Dalton Hubble	55bb4dfba6	Raise CoreDNS replica count to 2 or more * Run at least two replicas of CoreDNS to better support rolling updates (previously, kube-dns had a pod nanny) * On multi-master clusters, set the CoreDNS replica count to match the number of masters (e.g. a 3-master cluster previously used replicas:1, now replicas:3) * Add AntiAffinity preferred rule to favor distributing CoreDNS pods across controller nodes nodes	2018-10-13 20:31:29 -07:00
Dalton Hubble	43fe78a2cc	Raise scheduler/controller-manager replicas in multi-master * Continue to ensure scheduler and controller-manager run at least two replicas to support performing kubectl edits on single-master clusters (no change) * For multi-master clusters, set scheduler / controller-manager replica count to the number of masters (e.g. a 3-master cluster previously used replicas:2, now replicas:3)	2018-10-13 16:16:29 -07:00
Dalton Hubble	5a283b6443	Update etcd from v3.3.9 to v3.3.10 * https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10	2018-10-13 13:14:37 -07:00
Stephen Cuppett	ff0c271d7b	Fix Calico network policy docs link	2018-10-07 21:02:16 +02:00
Josh Reichardt	89088b6a99	Fix broken README links	2018-10-07 20:56:13 +02:00
Dalton Hubble	ee569e3a59	Change rooted links to absolute links for mkdocs v1.0 * mkdocs v1.0 now evaluates links beginning with / as absolute links that are only prepended with the site url	2018-10-07 20:51:54 +02:00
Dalton Hubble	daeaec0832	Fix requirements for docs builds	2018-10-07 19:26:55 +02:00
Dalton Hubble	db36036c81	Require terraform-provider-digitalocean plugin ~> 1.0 * Require a terraform-provider-digitalocean plugin version of 1.0 or higher within the same major version (e.g. allow 1.1 but not 2.0) * Change requirement from ~> 0.1.2 (which allowed up to but not including 1.0 release)	2018-10-02 17:09:19 +02:00
Dalton Hubble	7653e511be	Update CoreDNS and Calico versions * Update CoreDNS from 1.1.3 to 1.2.2 * Update Calico from v3.2.1 to v3.2.3	2018-10-02 16:07:48 +02:00
Dennis Schridde	f8474d68c9	Fix typo in maintenance docs	2018-09-25 03:07:00 -07:00
Dalton Hubble	032a24133b	Update Prometheus from v2.3.2 to v2.4.2 * https://github.com/prometheus/prometheus/releases/tag/v2.4.0 * https://github.com/prometheus/prometheus/releases/tag/v2.4.1 * https://github.com/prometheus/prometheus/releases/tag/v2.4.2	2018-09-21 22:27:11 -07:00
Dalton Hubble	f5f442b658	Improve the issue template prompt * Clarify that reporting versions as latest is not helpful	2018-09-15 10:05:07 -07:00
Dalton Hubble	ad871dbfa9	Update Kubernetes from v1.11.2 to v1.11.3 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113	2018-09-13 18:50:41 -07:00
Sendhil Panchadsaram	5801be53ff	Fix typo in ingress addon docs	2018-09-08 18:28:42 -07:00
Sendhil Panchadsaram	c1b1669cf8	Fix Google Cloud preemption link in docs	2018-09-08 18:28:25 -07:00
Dalton Hubble	dc03f7a4a9	Update nginx-ingress from 0.17.1 to 0.19.0 * If using --enable-ssl-passthrough or exposing TCP/UDP services, be aware of https://github.com/kubernetes/ingress-nginx/pull/3038 * Workarounds until the fix merges are to stay on 0.17.1, use the suggested development image, or revert to securityContext `runAsNonRoot: false` for a while (less secure)	2018-09-08 17:57:01 -07:00
Dalton Hubble	1b8234eb91	Update Grafana from v5.2.2 to v5.2.4 * https://github.com/grafana/grafana/releases/tag/v5.2.3 * https://github.com/grafana/grafana/releases/tag/v5.2.4	2018-09-08 15:41:20 -07:00
Dalton Hubble	4ba090feb0	Update kube-state-metrics from v1.3.1 to v1.4.0	2018-08-29 09:37:50 -07:00
Dalton Hubble	4882fe1053	Add docs for Azure Ingress and worker pools * Azure worker pools must be in the same region as the cluster itself unfortunately	2018-08-27 23:30:56 -07:00
Dalton Hubble	019009e9ee	Add outputs for Azure ingress IPv4 and worker pools	2018-08-27 23:30:32 -07:00
Dalton Hubble	991a5c6cee	Add new tutorial docs and links	2018-08-27 23:30:32 -07:00
Dalton Hubble	c60ec642bc	Fix Azure delete-node script to lowercase hostnames * Fix issue where worker nodes didn't delete themselves on scale-down or deallocation (e.g. low priority instances). Lowercase the hostname and delete the Kubernetes node * Kubelet registers the lowercase hostname as the node name, but Azure workers get hostname CLUSTER-worker-GENERATED where the generated identifier may contain uppercase characters	2018-08-27 23:30:32 -07:00
Dalton Hubble	38b4ff4700	Add module for Typhoon Azure with Container Linux	2018-08-27 23:30:32 -07:00
Dalton Hubble	7eb09237f4	Update Calico from v3.1.3 to v3.2.1 * Add new bird and felix readiness checks * Read MTU from ConfigMap veth_mtu * Add RBAC read for serviceaccounts * Remove invalid description from CRDs	2018-08-25 17:53:11 -07:00
Dalton Hubble	e58b424882	Fix firewall to allow etcd client traffic between controllers * Broaden internal-etcd firewall rule to allow etcd client traffic (2379) from other controller nodes * Previously, kube-apiservers were only able to connect to their node's local etcd peer. While master node outages were tolerated, reaching a healthy peer took longer than neccessary in some cases * Reduce time needed to bootstrap a cluster	2018-08-21 23:51:40 -07:00
Dalton Hubble	b8eeafe4f9	Template etcd_servers list to replace null_resource.repeat * Remove the last usage of null_resource.repeat, which has always been an eyesore for creating the etcd server list * Originally, #224 switched to templating the etcd_servers list for all clouds, but had to revert on GCP in #237 * https://github.com/poseidon/typhoon/pull/224 * https://github.com/poseidon/typhoon/pull/237	2018-08-21 22:46:24 -07:00
Dalton Hubble	bdf1e6986e	Fix terraform fmt	2018-08-21 21:59:55 -07:00
Dalton Hubble	99e3721181	Mention Fedora Atomic system container artifacts * Typhoon for Fedora Atomic uses system containers, container images containing metadata, but built directly from upstream and published and serve through Quay.io * https://github.com/poseidon/system-containers	2018-08-21 21:52:52 -07:00
Dalton Hubble	ea365b551a	Fix docs mentions of ELBs to NLBs * Typhoon AWS clusters use an NLB rather than an ELB, since v1.10.5 * Add a few missing links in CHANGES	2018-08-21 21:40:06 -07:00
Dalton Hubble	bbf2c13eef	Remove AWS security rule allowing ICMP packets to nodes * Deny ICMP packets for consistency across Typhoon clusters on various clouds and because there isn't much need to allow them	2018-08-21 21:16:16 -07:00
Dalton Hubble	da5d2c5321	Remove GCP firewall rule allowing Nginx Ingress health * Nginx Ingress addon no longer uses hostNework so Prometheus may scrape port 10254 via the CNI network, rather than via the host address	2018-08-21 21:06:03 -07:00
Dalton Hubble	bceec9fdf5	Sort firewall / security rules and add comments * No functional changes to network firewalls	2018-08-21 20:53:16 -07:00
Becca Powell	49a9dc9b8b	Fix typo in Prometheus alerting rules	2018-08-21 16:55:49 -07:00
Dalton Hubble	bec5250e73	Remove unofficial bare-metal _networkds variables Remove controller_networkds and worker_networkds variables. These variables were always listed as experimental, unsupported, and excluded from documentation in anticipation of Container Linux Config snippets * Use Container Linux Config snippets on bare-metal instead. They provide safer, more powerful, and more elegant host customization	2018-08-13 23:33:29 -07:00