Update changelog for v1.13.1 release

Update terraform-provider-ct plugin from v0.2.1 to v0.3.0
* Provide migration instructions for upgrading terraform-provider-ct in-place for v1.12.2+ clusters * Require switching from ~/.terraformrc to the Terraform third-party plugins directory ~/.terraform.d/plugins/ * Require Container Linux 1688.5.3 or newer
2025-08-02 13:21:34 +02:00 · 2018-12-17 14:28:27 -08:00 · 2018-12-17 14:13:50 -08:00 · 2018-12-16 10:47:59 -08:00 · 2018-12-15 22:52:28 -08:00 · 2018-12-15 18:05:16 -08:00
115 changed files with 1024 additions and 635 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@ -5,8 +5,8 @@
 ### Environment

 * Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, fedora-atomic
-* Ref: Release version or Git SHA (reporting latest is **not** helpful)
+* OS: container-linux, flatcar-linux, or fedora-atomic
+* Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
 * Terraform: `terraform version` (reporting latest is **not** helpful)
 * Plugins: Provider plugin versions (reporting latest is **not** helpful)

--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,20 +4,123 @@ Notable changes between versions.

 ## Latest

-* Kubernetes [v1.12.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1121)
-* Update etcd from v3.3.9 to [v3.3.10](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10)
-* Update CoreDNS from 1.1.3 to 1.2.2
-* Update Calico from v3.2.1 to v3.2.3
-* On multi-controller clusters, raise scheduler and controller-manager replics to equal the number of controller nodes ([#312](https://github.com/poseidon/typhoon/pull/312))
-  * Single-controller clusters continue to run 2 replicas as before
-* Raise default CoreDNS replica count to the larger of 2 or the number of controller nodes ([#313](https://github.com/poseidon/typhoon/pull/313))
-  * Add AntiAffinity preferred rule to favor spreading CoreDNS pods
-* Annotate Kubernetes control plane and addons to start containers with the Docker runtime's default seccomp profile ([#319](https://github.com/poseidon/typhoon/pull/319))
-  * Override Kubernetes default behavior that starts containers with seccomp=unconfined
+## v1.13.1
+
+* Kubernetes [v1.13.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1131)
+* Update Calico from v3.3.2 to [v3.4.0](https://docs.projectcalico.org/v3.4/releases/) ([#362](https://github.com/poseidon/typhoon/pull/362))
+  * Install CNI plugins with an init container rather than a sidecar
+  * Improve the `calico-node` ClusterRole
+* Recommend updating `terraform-provider-ct` plugin from v0.2.1 to v0.3.0 ([#363](https://github.com/poseidon/typhoon/pull/363))
+  * [Migration](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct) instructions for upgrading `terraform-provider-ct` in-place for v1.12.2+ clusters (**action required**)
+  * [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`
+  * Require Container Linux 1688.5.3 or newer
+
+#### Google Cloud
+
+* Increase TCP proxy apiserver backend service timeout from 1 minute to 5 minutes ([#361](https://github.com/poseidon/typhoon/pull/361))
+  * Align `port-forward` behavior closer to AWS/Azure (no timeout)
+
+#### Addons
+
+* Update Grafana from v5.4.0 to v5.4.2
+
+## v1.13.0
+
+* Kubernetes [v1.13.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1130)
+* Update Calico from v3.3.1 to [v3.3.2](https://docs.projectcalico.org/v3.3/releases/)
+
+#### Addons
+
+* Update Grafana from v5.3.4 to v5.4.0
+* Disable Grafana login form, since admin user can't be disabled ([#352](https://github.com/poseidon/typhoon/pull/352))
+  * Example manifests aim to provide a read-only dashboard view
+
+## v1.12.3
+
+* Kubernetes [v1.12.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1123)
+* Add `enable_reporting` variable (default "false") to provide upstreams with usage data ([#345](https://github.com/poseidon/typhoon/pull/345))
+* Change kube-apiserver `--kubelet-preferred-address-types` to InternalIP,ExternalIP,Hostname
+* Update Calico from v3.3.0 to [v3.3.1](https://docs.projectcalico.org/v3.3/releases/)
+  * Disable Felix usage reporting by default ([#345](https://github.com/poseidon/typhoon/pull/345))
+* Improve flannel manifests
+  * [Rename](https://github.com/poseidon/terraform-render-bootkube/commit/d045a8e6b8eccfbb9d69bb51953b5a93d23f67f7) `kube-flannel` DaemonSet to `flannel` and `kube-flannel-cfg` ConfigMap to `flannel-config` 
+  * [Drop](https://github.com/poseidon/terraform-render-bootkube/commit/39f9afb3360ec642e5b98457c8bd07eda35b6c96) unused mounts and add a CPU resource request
+* Update CoreDNS from v1.2.4 to [v1.2.6](https://coredns.io/2018/11/05/coredns-1.2.6-release/)
+  * Enable CoreDNS `loop` and `loadbalance` plugins ([#340](https://github.com/poseidon/typhoon/pull/340))
+* Fix pod-checkpointer log noise and checkpointable pods detection ([#346](https://github.com/poseidon/typhoon/pull/346))
+* Use kubernetes-incubator/bootkube v0.14.0
+* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`.
+  * Allows pinning `terraform-provider-ct` and `terraform-provider-matchbox` versions
+  * Improves safety of later plugin version migrations

 #### Azure

-* Remove admin_password field (disabled) since it is now optional
+* Use eviction policy `Delete` for `Low` priority virtual machine scale set workers ([#343](https://github.com/poseidon/typhoon/pull/343))
+  * Fix issue where Azure defaults to `Deallocate` eviction policy, which required manually restarting deallocated instances. `Delete` policy aligns Azure with AWS and GCP behavior.
+  * Require `terraform-provider-azurerm` v1.19+ (action required)
+
+#### Bare-Metal
+
+* Add Kubelet `/etc/iscsi` and `iscsadm` mounts on bare-metal for iSCSI ([#103](https://github.com/poseidon/typhoon/pull/103))
+
+#### Addons
+
+* Update nginx-ingress from v0.20.0 to v0.21.0
+* Update Prometheus from v2.4.3 to v2.5.0
+* Update Grafana from v5.3.2 to v5.3.4
+
+## v1.12.2
+
+* Kubernetes [v1.12.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122)
+* Update CoreDNS from 1.2.2 to [1.2.4](https://github.com/coredns/coredns/releases/tag/v1.2.4)
+* Update Calico from v3.2.3 to [v3.3.0](https://docs.projectcalico.org/v3.3/releases/)
+* Disable Kubelet read-only port ([#324](https://github.com/poseidon/typhoon/pull/324))
+* Fix CoreDNS AntiAffinity spec to prefer spreading replicas
+* Ignore controller node user-data changes ([#335](https://github.com/poseidon/typhoon/pull/335))
+  * Once all managed clusters use v1.12.2, it is possible to update `terraform-provider-ct`
+
+#### AWS
+
+* Add `disk_iops` variable for EBS volume IOPS ([#314](https://github.com/poseidon/typhoon/pull/314))
+
+#### Azure
+
+* Use new `azurerm_network_interface_backend_address_pool_association` ([#332](https://github.com/poseidon/typhoon/pull/332))
+  * Require `terraform-provider-azurerm` v1.17+ (action required)
+* Add `primary` field to `ip_configuration` needed by v1.17+ ([#331](https://github.com/poseidon/typhoon/pull/331))
+
+#### DigitalOcean
+
+* Add AAAA DNS records resolving to worker nodes ([#333](https://github.com/poseidon/typhoon/pull/333))
+  * Hosting IPv6 apps requires editing nginx-ingress with `hostNetwork: true`
+
+#### Google Cloud
+
+* Add an IPv6 address and IPv6 forwarding rules for load balancing IPv6 Ingress ([#334](https://github.com/poseidon/typhoon/pull/334))
+  * Add `ingress_static_ipv6` output variable for use in AAAA DNS records
+  * Allow serving IPv6 applications via Kubernetes Ingress
+
+#### Addons
+
+* Configure Heapster to scrape Kubelets with bearer token auth ([#323](https://github.com/poseidon/typhoon/pull/323))
+* Update Grafana from v5.3.1 to v5.3.2
+
+## v1.12.1
+
+* Kubernetes [v1.12.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1121)
+* Update etcd from v3.3.9 to [v3.3.10](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10)
+* Update CoreDNS from 1.1.3 to [1.2.2](https://github.com/coredns/coredns/releases/tag/v1.2.2)
+* Update Calico from v3.2.1 to [v3.2.3](https://docs.projectcalico.org/v3.2/releases/)
+* Raise scheduler and controller-manager replicas to the larger of 2 or the number of controller nodes ([#312](https://github.com/poseidon/typhoon/pull/312))
+  * Single-controller clusters continue to run 2 replicas as before
+* Raise default CoreDNS replicas to the larger of 2 or the number of controller nodes ([#313](https://github.com/poseidon/typhoon/pull/313))
+  * Add AntiAffinity preferred rule to favor spreading CoreDNS pods
+* Annotate control plane and addon containers to use the Docker runtime seccomp profile ([#319](https://github.com/poseidon/typhoon/pull/319))
+  * Override Kubernetes default behavior that starts containers with `seccomp=unconfined`
+
+#### Azure
+
+* Remove `admin_password` field (disabled) since it is now optional
  * Require `terraform-provider-azurerm` v1.16+ (action required)

 #### Bare-Metal
--- a/README.md
+++ b/README.md
@ -11,29 +11,32 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)

 ## Modules

-Typhoon provides a Terraform Module for each supported operating system and platform.
+Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
-| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |

-The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
+Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
+| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
+| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |

 ## Documentation

@ -47,7 +50,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.1"
  
  providers = {
    google   = "google.default"
@ -87,10 +90,10 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 ```sh
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.1
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.1
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.1
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.1
 ```

 List the pods.
@ -102,6 +105,7 @@ kube-system   calico-node-1cs8z                         2/2    Running   0
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
@ -111,6 +115,7 @@ kube-system   kube-proxy-njn47                          1/1    Running   0
 kube-system   kube-scheduler-3895335239-5x87r           1/1    Running   0         6m
 kube-system   kube-scheduler-3895335239-bzrrt           1/1    Running   1         6m
 kube-system   pod-checkpointer-l6lrt                    1/1    Running   0         6m
+kube-system   pod-checkpointer-l6lrt-controller-0       1/1    Running   0         6m
 ```

 ## Non-Goals
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -23,12 +23,14 @@ spec:
    spec:
      containers:
        - name: grafana
-          image: grafana/grafana:5.3.1
+          image: grafana/grafana:5.4.2
          env:
            - name: GF_SERVER_HTTP_PORT
              value: "8080"
            - name: GF_AUTH_BASIC_ENABLED
              value: "false"
+            - name: GF_AUTH_DISABLE_LOGIN_FORM
+              value: "true"
            - name: GF_AUTH_ANONYMOUS_ENABLED
              value: "true"
            - name: GF_AUTH_ANONYMOUS_ORG_ROLE
--- a/addons/heapster/cluster-role-binding.yaml
+++ b/addons/heapster/cluster-role-binding.yaml
@ -5,7 +5,7 @@ metadata:
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
-  name: system:heapster
+  name: heapster
 subjects:
 - kind: ServiceAccount
  name: heapster
--- a/addons/heapster/cluster-role.yaml
+++ b/addons/heapster/cluster-role.yaml
@ -0,0 +1,30 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: heapster
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - events
+  - namespaces
+  - nodes
+  - pods
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - deployments
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - nodes/stats
+  verbs:
+  - get
--- a/addons/heapster/deployment.yaml
+++ b/addons/heapster/deployment.yaml
@ -23,7 +23,7 @@ spec:
          image: k8s.gcr.io/heapster-amd64:v1.5.4
          command:
            - /heapster
-            - --source=kubernetes.summary_api:''
+            - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
          livenessProbe:
            httpGet:
              path: /healthz
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/prometheus/config.yaml
+++ b/addons/prometheus/config.yaml
@ -102,7 +102,7 @@ data:
          regex: 'true'
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
-        - source_labels: [__meta_kubernetes_node_name]
+        - source_labels: [__meta_kubernetes_node_address_InternalIP]
          action: replace
          target_label: __address__
          replacement: '${1}:2381'
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -20,7 +20,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.4.3
+          image: quay.io/prometheus/prometheus:v2.5.0
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/aws/container-linux/kubernetes/bootkube.tf
+++ b/aws/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@ -11,4 +11,5 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }
--- a/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -88,6 +88,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -122,7 +123,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -142,17 +143,14 @@ storage:
          set -e
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
            --mount volume=assets,target=/assets \
            --volume bootstrap,kind=host,source=/etc/kubernetes \
            --mount volume=bootstrap,target=/etc/kubernetes \
            $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
            --net=host \
            --dns=host \
            --exec=/bootkube -- start --asset-dir=/assets "$@"
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -24,12 +24,13 @@ resource "aws_instance" "controllers" {
  instance_type = "${var.controller_type}"

  ami       = "${local.ami_id}"
-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"

  # storage
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
@ -38,12 +39,23 @@ resource "aws_instance" "controllers" {
  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]

  lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
  }
 }

-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
  count = "${var.controller_count}"

  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@ -73,10 +85,3 @@ data "template_file" "etcds" {
    dns_zone     = "${var.dns_zone}"
  }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
-}
--- a/aws/container-linux/kubernetes/outputs.tf
+++ b/aws/container-linux/kubernetes/outputs.tf
@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
+
 # Outputs for Kubernetes Ingress

 output "ingress_dns_name" {
--- a/aws/container-linux/kubernetes/security.tf
+++ b/aws/container-linux/kubernetes/security.tf
@ -104,27 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
  self      = true
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
  security_group_id = "${aws_security_group.controller.id}"

@ -300,27 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
  self      = true
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
  security_group_id = "${aws_security_group.worker.id}"

--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -59,6 +59,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
  type        = "string"
  default     = ""
@ -128,3 +134,9 @@ variable "cluster_domain_suffix" {
  type        = "string"
  default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -60,6 +60,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -92,7 +93,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -110,7 +111,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.1 \
+            docker://k8s.gcr.io/hyperkube:v1.13.1 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -52,6 +52,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
  type        = "string"
  default     = ""
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -46,12 +46,13 @@ resource "aws_launch_configuration" "worker" {
  spot_price        = "${var.spot_price}"
  enable_monitoring = false

-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"

  # storage
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
@ -64,8 +65,15 @@ resource "aws_launch_configuration" "worker" {
  }
 }

-# Worker Container Linux Config
-data "template_file" "worker_config" {
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.clc_snippets}"]
+}
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars = {
@ -75,9 +83,3 @@ data "template_file" "worker_config" {
    cluster_domain_suffix = "${var.cluster_domain_suffix}"
  }
 }
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
-  pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
-}
--- a/aws/fedora-atomic/kubernetes/README.md
+++ b/aws/fedora-atomic/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/aws/fedora-atomic/kubernetes/bootkube.tf
+++ b/aws/fedora-atomic/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@ -11,6 +11,7 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"

  # Fedora
  trusted_certs_dir = "/etc/pki/tls/certs"
--- a/aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -19,24 +19,9 @@ write_files:
      ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
      ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
      ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/latest/meta-data/local-ipv4\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
    content: |
      [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
      Wants=rpc-statd.service
      [Service]
      ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -65,6 +50,7 @@ write_files:
        --node-labels=node-role.kubernetes.io/master \
        --node-labels=node-role.kubernetes.io/controller="true" \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/kubernetes/kubeconfig
@ -93,10 +79,9 @@ runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.1"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
  - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, cloud-metadata.service]
  - [systemctl, start, --no-block, kubelet.service]
 users:
  - default
--- a/aws/fedora-atomic/kubernetes/controllers.tf
+++ b/aws/fedora-atomic/kubernetes/controllers.tf
@ -30,6 +30,7 @@ resource "aws_instance" "controllers" {
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
@ -38,7 +39,10 @@ resource "aws_instance" "controllers" {
  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]

  lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
  }
 }

--- a/aws/fedora-atomic/kubernetes/outputs.tf
+++ b/aws/fedora-atomic/kubernetes/outputs.tf
@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
+
 # Outputs for Kubernetes Ingress

 output "ingress_dns_name" {
--- a/aws/fedora-atomic/kubernetes/security.tf
+++ b/aws/fedora-atomic/kubernetes/security.tf
@ -104,27 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
  self      = true
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
  security_group_id = "${aws_security_group.controller.id}"

@ -300,27 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
  self      = true
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
  security_group_id = "${aws_security_group.worker.id}"

--- a/aws/fedora-atomic/kubernetes/variables.tf
+++ b/aws/fedora-atomic/kubernetes/variables.tf
@ -53,6 +53,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
  type        = "string"
  default     = ""
@ -110,3 +116,9 @@ variable "cluster_domain_suffix" {
  type        = "string"
  default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl
+++ b/aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl
@ -1,23 +1,8 @@
 #cloud-config
 write_files:
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/latest/meta-data/local-ipv4\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
    content: |
      [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
      Wants=rpc-statd.service
      [Service]
      ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -43,6 +28,7 @@ write_files:
        --network-plugin=cni \
        --node-labels=node-role.kubernetes.io/node \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/kubernetes/kubeconfig
    permissions: '0644'
@ -68,8 +54,7 @@ bootcmd:
 runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
-  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.1"
  - [systemctl, start, --no-block, kubelet.service]
 users:
  - default
--- a/aws/fedora-atomic/kubernetes/workers/variables.tf
+++ b/aws/fedora-atomic/kubernetes/workers/variables.tf
@ -46,6 +46,12 @@ variable "disk_type" {
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
  type        = "string"
  default     = ""
--- a/aws/fedora-atomic/kubernetes/workers/workers.tf
+++ b/aws/fedora-atomic/kubernetes/workers/workers.tf
@ -52,6 +52,7 @@ resource "aws_launch_configuration" "worker" {
  root_block_device {
    volume_type = "${var.disk_type}"
    volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
  }

  # network
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/azure/container-linux/kubernetes/bootkube.tf
+++ b/azure/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@ -10,4 +10,5 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }
--- a/azure/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/azure/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -88,6 +88,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -122,7 +123,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -142,17 +143,14 @@ storage:
          set -e
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
            --mount volume=assets,target=/assets \
            --volume bootstrap,kind=host,source=/etc/kubernetes \
            --mount volume=bootstrap,target=/etc/kubernetes \
            $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
            --net=host \
            --dns=host \
            --exec=/bootkube -- start --asset-dir=/assets "$@"
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -85,6 +85,7 @@ resource "azurerm_virtual_machine" "controllers" {
  lifecycle {
    ignore_changes = [
      "storage_os_disk",
+      "os_profile",
    ]
  }
 }
@ -105,12 +106,16 @@ resource "azurerm_network_interface" "controllers" {

    # public IPv4
    public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
-
-    # backend address pool to which the NIC should be added
-    load_balancer_backend_address_pools_ids = ["${azurerm_lb_backend_address_pool.controller.id}"]
  }
 }

+# Add controller NICs to the controller backend address pool
+resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+  network_interface_id    = "${azurerm_network_interface.controllers.id}"
+  ip_configuration_name   = "ip0"
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+}
+
 # Controller public IPv4 addresses
 resource "azurerm_public_ip" "controllers" {
  count               = "${var.controller_count}"
--- a/azure/container-linux/kubernetes/outputs.tf
+++ b/azure/container-linux/kubernetes/outputs.tf
@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
+
 # Outputs for Kubernetes Ingress

 output "ingress_static_ipv4" {
--- a/azure/container-linux/kubernetes/require.tf
+++ b/azure/container-linux/kubernetes/require.tf
@ -5,7 +5,7 @@ terraform {
 }

 provider "azurerm" {
-  version = "~> 1.16"
+  version = "~> 1.19"
 }

 provider "local" {
@ -23,4 +23,3 @@ provider "template" {
 provider "tls" {
  version = "~> 1.0"
 }
-
--- a/azure/container-linux/kubernetes/security.tf
+++ b/azure/container-linux/kubernetes/security.tf
@ -117,22 +117,6 @@ resource "azurerm_network_security_rule" "controller-kubelet" {
  destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "azurerm_network_security_rule" "controller-kubelet-read" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
-
-  name                        = "allow-kubelet-read"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
-  priority                    = "2035"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10255"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
-}
-
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules

@ -269,22 +253,6 @@ resource "azurerm_network_security_rule" "worker-kubelet" {
  destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "azurerm_network_security_rule" "worker-kubelet-read" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
-
-  name                        = "allow-kubelet-read"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
-  priority                    = "2030"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10255"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
-}
-
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules

--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -115,3 +115,9 @@ variable "cluster_domain_suffix" {
  type        = "string"
  default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -60,6 +60,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -92,7 +93,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -110,7 +111,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.1 \
+            docker://k8s.gcr.io/hyperkube:v1.13.1 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
--- a/azure/container-linux/kubernetes/workers/ingress.tf
+++ b/azure/container-linux/kubernetes/workers/ingress.tf
@ -1 +0,0 @@
-
--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -37,7 +37,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
  os_profile {
    computer_name_prefix = "${var.name}-worker-"
    admin_username       = "core"
-    custom_data          = "${element(data.ct_config.worker-ignitions.*.rendered, count.index)}"
+    custom_data          = "${data.ct_config.worker-ignition.rendered}"
  }

  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@ -58,6 +58,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {

    ip_configuration {
      name      = "ip0"
+      primary   = true
      subnet_id = "${var.subnet_id}"

      # backend address pool to which the NIC should be added
@ -66,8 +67,9 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
  }

  # lifecycle
-  priority            = "${var.priority}"
  upgrade_policy_mode = "Manual"
+  priority            = "${var.priority}"
+  eviction_policy     = "Delete"
 }

 # Scale up or down to maintain desired number, tolerating deallocations.
@ -93,14 +95,14 @@ resource "azurerm_autoscale_setting" "workers" {
 }

 # Worker Ignition configs
-data "ct_config" "worker-ignitions" {
-  content      = "${data.template_file.worker-configs.rendered}"
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
  pretty_print = false
  snippets     = ["${var.clc_snippets}"]
 }

 # Worker Container Linux configs
-data "template_file" "worker-configs" {
+data "template_file" "worker-config" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars = {
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/bare-metal/container-linux/kubernetes/bootkube.tf
+++ b/bare-metal/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name                    = "${var.cluster_name}"
  api_servers                     = ["${var.k8s_domain_name}"]
@ -12,4 +12,5 @@ module "bootkube" {
  pod_cidr                        = "${var.pod_cidr}"
  service_cidr                    = "${var.service_cidr}"
  cluster_domain_suffix           = "${var.cluster_domain_suffix}"
+  enable_reporting                = "${var.enable_reporting}"
 }
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -70,6 +70,10 @@ systemd:
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
          --volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
          --insecure-options=image"
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -97,6 +101,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -123,7 +128,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/hostname
      filesystem: root
      mode: 0644
@ -149,17 +154,14 @@ storage:
          set -e
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
-            --volume assets,kind=host,source=$BOOTKUBE_ASSETS \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
            --mount volume=assets,target=/assets \
            --volume bootstrap,kind=host,source=/etc/kubernetes \
            --mount volume=bootstrap,target=/etc/kubernetes \
            $$RKT_OPTS \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
            --net=host \
            --dns=host \
            --exec=/bootkube -- start --asset-dir=/assets "$@"
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -45,6 +45,10 @@ systemd:
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
          --volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
          --insecure-options=image"
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@ -69,6 +73,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -84,7 +89,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/kubernetes/outputs.tf
+++ b/bare-metal/container-linux/kubernetes/outputs.tf
@ -1,3 +1,7 @@
 output "kubeconfig" {
  value = "${module.bootkube.kubeconfig}"
 }
+
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
--- a/bare-metal/container-linux/kubernetes/variables.tf
+++ b/bare-metal/container-linux/kubernetes/variables.tf
@ -141,3 +141,9 @@ variable "kernel_args" {
  type        = "list"
  default     = []
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/bare-metal/fedora-atomic/kubernetes/README.md
+++ b/bare-metal/fedora-atomic/kubernetes/README.md
@ -11,8 +11,8 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

--- a/bare-metal/fedora-atomic/kubernetes/bootkube.tf
+++ b/bare-metal/fedora-atomic/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${var.k8s_domain_name}"]
@ -11,6 +11,7 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"

  # Fedora
  trusted_certs_dir = "/etc/pki/tls/certs"
--- a/bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -51,6 +51,7 @@ write_files:
        --node-labels=node-role.kubernetes.io/master \
        --node-labels=node-role.kubernetes.io/controller="true" \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
@ -84,8 +85,8 @@ runcmd:
  - [systemctl, restart, NetworkManager]
  - [hostnamectl, set-hostname, ${domain_name}]
  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.1"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
  - [systemctl, start, --no-block, etcd.service]
  - [systemctl, enable, kubelet.path]
  - [systemctl, start, --no-block, kubelet.path]
--- a/bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
+++ b/bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
@ -29,6 +29,7 @@ write_files:
        --network-plugin=cni \
        --node-labels=node-role.kubernetes.io/node \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
    content: |
@ -59,7 +60,7 @@ runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, restart, NetworkManager]
  - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.1"
  - [systemctl, enable, kubelet.path]
  - [systemctl, start, --no-block, kubelet.path]
 users:
--- a/bare-metal/fedora-atomic/kubernetes/outputs.tf
+++ b/bare-metal/fedora-atomic/kubernetes/outputs.tf
@ -1,3 +1,8 @@
 output "kubeconfig" {
  value = "${module.bootkube.kubeconfig}"
 }
+
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
+
--- a/bare-metal/fedora-atomic/kubernetes/variables.tf
+++ b/bare-metal/fedora-atomic/kubernetes/variables.tf
@ -110,3 +110,9 @@ variable "kernel_args" {
  type        = "list"
  default     = []
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,10 +11,11 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, and other [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

--- a/digital-ocean/container-linux/kubernetes/bootkube.tf
+++ b/digital-ocean/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@ -11,4 +11,5 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -56,12 +56,9 @@ systemd:
      contents: |
        [Unit]
        Description=Kubelet via Hyperkube
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
-        EnvironmentFile=/run/metadata/coreos
        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
          --volume=resolv,kind=host,source=/etc/resolv.conf \
          --mount volume=resolv,target=/etc/resolv.conf \
@ -93,13 +90,13 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/master \
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@ -128,7 +125,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -148,17 +145,14 @@ storage:
          set -e
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
            --mount volume=assets,target=/assets \
            --volume bootstrap,kind=host,source=/etc/kubernetes \
            --mount volume=bootstrap,target=/etc/kubernetes \
            $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
            --net=host \
            --dns=host \
            --exec=/bootkube -- start --asset-dir=/assets "$@"
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -31,12 +31,9 @@ systemd:
      contents: |
        [Unit]
        Description=Kubelet via Hyperkube
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
-        EnvironmentFile=/run/metadata/coreos
        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
          --volume=resolv,kind=host,source=/etc/resolv.conf \
          --mount volume=resolv,target=/etc/resolv.conf \
@ -66,12 +63,12 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -98,7 +95,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -116,7 +113,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.1 \
+            docker://k8s.gcr.io/hyperkube:v1.13.1 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/controllers.tf
+++ b/digital-ocean/container-linux/kubernetes/controllers.tf
@ -44,12 +44,18 @@ resource "digitalocean_droplet" "controllers" {
  ipv6               = true
  private_networking = true

-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
  ssh_keys  = ["${var.ssh_fingerprints}"]

  tags = [
    "${digitalocean_tag.controllers.id}",
  ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }

 # Tag to label controllers
@ -57,8 +63,16 @@ resource "digitalocean_tag" "controllers" {
  name = "${var.cluster_name}-controller"
 }

-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
  count = "${var.controller_count}"

  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@ -85,11 +99,3 @@ data "template_file" "etcds" {
    dns_zone     = "${var.dns_zone}"
  }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-
-  snippets = ["${var.controller_clc_snippets}"]
-}
--- a/digital-ocean/container-linux/kubernetes/outputs.tf
+++ b/digital-ocean/container-linux/kubernetes/outputs.tf
@ -1,9 +1,14 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
+
 output "controllers_dns" {
  value = "${digitalocean_record.controllers.0.fqdn}"
 }

 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }

 output "controllers_ipv4" {
--- a/digital-ocean/container-linux/kubernetes/variables.tf
+++ b/digital-ocean/container-linux/kubernetes/variables.tf
@ -92,3 +92,9 @@ variable "cluster_domain_suffix" {
  type        = "string"
  default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/digital-ocean/container-linux/kubernetes/workers.tf
+++ b/digital-ocean/container-linux/kubernetes/workers.tf
@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
  count = "${var.worker_count}"

  # DNS zone where record should be created
@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
  value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }

+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
  count = "${var.worker_count}"
@ -25,12 +37,16 @@ resource "digitalocean_droplet" "workers" {
  ipv6               = true
  private_networking = true

-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"
  ssh_keys  = ["${var.ssh_fingerprints}"]

  tags = [
    "${digitalocean_tag.workers.id}",
  ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }

 # Tag to label workers
@ -38,8 +54,15 @@ resource "digitalocean_tag" "workers" {
  name = "${var.cluster_name}-worker"
 }

-# Worker Container Linux Config
-data "template_file" "worker_config" {
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
+  pretty_print = false
+  snippets     = ["${var.worker_clc_snippets}"]
+}
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars = {
@ -47,9 +70,3 @@ data "template_file" "worker_config" {
    cluster_domain_suffix = "${var.cluster_domain_suffix}"
  }
 }
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
-  pretty_print = false
-  snippets     = ["${var.worker_clc_snippets}"]
-}
--- a/digital-ocean/fedora-atomic/kubernetes/README.md
+++ b/digital-ocean/fedora-atomic/kubernetes/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/digital-ocean/fedora-atomic/kubernetes/bootkube.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@ -11,6 +11,7 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"

  # Fedora
  trusted_certs_dir = "/etc/pki/tls/certs"
--- a/digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -19,24 +19,9 @@ write_files:
      ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
      ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
      ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
    content: |
      [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
      Wants=rpc-statd.service
      [Service]
      ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -65,6 +50,7 @@ write_files:
        --node-labels=node-role.kubernetes.io/master \
        --node-labels=node-role.kubernetes.io/controller="true" \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
@ -90,10 +76,9 @@ bootcmd:
 runcmd:
  - [systemctl, daemon-reload]
  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.1"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
  - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, cloud-metadata.service]
  - [systemctl, enable, kubelet.path]
  - [systemctl, start, --no-block, kubelet.path]
 users:
--- a/digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
+++ b/digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl
@ -1,23 +1,8 @@
 #cloud-config
 write_files:
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
    content: |
      [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
      Wants=rpc-statd.service
      [Service]
      ExecStartPre=/bin/mkdir -p /opt/cni/bin
@ -43,6 +28,7 @@ write_files:
        --network-plugin=cni \
        --node-labels=node-role.kubernetes.io/node \
        --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
  - path: /etc/systemd/system/kubelet.path
    content: |
@ -65,8 +51,7 @@ bootcmd:
  - [modprobe, ip_vs]
 runcmd:
  - [systemctl, daemon-reload]
-  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.1"
  - [systemctl, enable, kubelet.path]
  - [systemctl, start, --no-block, kubelet.path]
 users:
--- a/digital-ocean/fedora-atomic/kubernetes/controllers.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/controllers.tf
@ -50,6 +50,12 @@ resource "digitalocean_droplet" "controllers" {
  tags = [
    "${digitalocean_tag.controllers.id}",
  ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }

 # Tag to label controllers
--- a/digital-ocean/fedora-atomic/kubernetes/outputs.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/outputs.tf
@ -1,9 +1,14 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.user-kubeconfig}"
+}
+
 output "controllers_dns" {
  value = "${digitalocean_record.controllers.0.fqdn}"
 }

 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }

 output "controllers_ipv4" {
--- a/digital-ocean/fedora-atomic/kubernetes/variables.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/variables.tf
@ -85,3 +85,9 @@ variable "cluster_domain_suffix" {
  type        = "string"
  default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}
--- a/digital-ocean/fedora-atomic/kubernetes/workers.tf
+++ b/digital-ocean/fedora-atomic/kubernetes/workers.tf
@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
  count = "${var.worker_count}"

  # DNS zone where record should be created
@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
  value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }

+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
  count = "${var.worker_count}"
@ -31,6 +43,10 @@ resource "digitalocean_droplet" "workers" {
  tags = [
    "${digitalocean_tag.workers.id}",
  ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }

 # Tag to label workers
--- a/docs/addons/ingress.md
+++ b/docs/addons/ingress.md
@ -4,7 +4,7 @@ Nginx Ingress controller pods accept and demultiplex HTTP, HTTPS, TCP, or UDP tr

 ## AWS

-On AWS, a network load balancer (NLB) distributes traffic across a target group of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.
+On AWS, a network load balancer (NLB) distributes TCP traffic across two target groups (port 80 and 443) of worker nodes running an Ingress controller deployment. Security groups rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.

 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.

@ -37,7 +37,7 @@ resource "google_dns_record_set" "some-application" {

 ## Azure

-On Azure, a load balancer distributes traffic across a backend pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
+On Azure, a load balancer distributes traffic across a backend address pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.

 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.

@ -101,7 +101,7 @@ resource "google_dns_record_set" "some-application" {

 ## Digital Ocean

-On Digital Ocean, a DNS A record (e.g. `nemo-workers.example.com`) resolves to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
+On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.

 Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, default backend, and namespace.

@ -124,11 +124,14 @@ resource "google_dns_record_set" "some-application" {
 }
 ```

+!!! note
+    Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
+
 [^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.

 ## Google Cloud

-On Google Cloud, a TCP Proxy load balancer distributes traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.
+On Google Cloud, a TCP Proxy load balancer distributes IPv4 and IPv6 TCP traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.

 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.

@ -136,7 +139,7 @@ Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, de
 kubectl apply -R -f addons/nginx-ingress/google-cloud
 ```

-For each application, add a DNS record resolving to the load balancer's IPv4 address.
+For each application, add DNS A records resolving to the load balancer's IPv4 address and DNS AAAA records resolving to the load balancer's IPv6 address.

 ```
 app1.example.com -> 11.22.33.44
@ -144,10 +147,10 @@ app2.example.com -> 11.22.33.44
 app3.example.com -> 11.22.33.44
 ```

-Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's output `ingress_static_ipv4`. For example, you might use Terraform to manage a Google Cloud DNS record:
+Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's outputs `ingress_static_ipv4` and `ingress_static_ipv6`. For example, you might use Terraform to manage a Google Cloud DNS record:

 ```tf
-resource "google_dns_record_set" "some-application" {
+resource "google_dns_record_set" "app-record-a" {
  # DNS zone name
  managed_zone = "example-zone"

@ -157,4 +160,15 @@ resource "google_dns_record_set" "some-application" {
  ttl     = 300
  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv4}"]
 }
+
+resource "google_dns_record_set" "app-record-aaaa" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "AAAA"
+  ttl     = 300
+  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv6}"]
+}
 ```
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -16,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo

 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.13.1"
  
  providers = {
    aws = "aws.default"
@ -82,7 +82,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define

 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.13.1"
  
  providers = {
    azurerm = "azurerm.default"
@ -152,7 +152,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.13.1"

  providers = {
    google = "google.default"
@ -187,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.12.1
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.12.1
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.12.1
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.13.1
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.13.1
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.13.1
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.13.1
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.13.1
 ```

 ### Variables
--- a/docs/architecture/aws.md
+++ b/docs/architecture/aws.md
@ -0,0 +1,13 @@
+# AWS
+
+## IPv6
+
+Status of IPv6 on Typhoon AWS clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | No        |
+
+* AWS Network Load Balancers do not support `dualstack`.
--- a/docs/architecture/azure.md
+++ b/docs/architecture/azure.md
@ -0,0 +1,13 @@
+# Azure
+
+## IPv6
+
+Status of IPv6 on Typhoon Azure clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | No        |
+| Node Outbound IPv6      | No        |
+| Kubernetes Ingress IPv6 | No        |
+
+* Azure does not allow reserving a static IPv6 address 
--- a/docs/architecture/bare-metal.md
+++ b/docs/architecture/bare-metal.md
@ -0,0 +1,13 @@
+# Bare-Metal
+
+## IPv6
+
+Status of IPv6 on Typhoon bare-metal clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Possible  |
+
+IPv6 support depends upon the bare-metal network environment.
--- a/docs/architecture/concepts.md
+++ b/docs/architecture/concepts.md
@ -69,7 +69,7 @@ Module versioning ensures `terraform get --update` only fetches the desired vers

 Maintain Terraform configs for "live" infrastructure in a versioned repository. Seek to organize configs to reflect resources that should be managed together in a `terraform apply` invocation.

-You may choose to organize resources all together, by team, by project, or some other scheme. Here's an example that manages four clusters together:
+You may choose to organize resources all together, by team, by project, or some other scheme. Here's an example that manages clusters together:

 ```sh
 .git/
--- a/docs/architecture/digitalocean.md
+++ b/docs/architecture/digitalocean.md
@ -0,0 +1,11 @@
+# AWS
+
+## IPv6
+
+Status of IPv6 on Typhoon DigitalOcean clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Possible  |
--- a/docs/architecture/google-cloud.md
+++ b/docs/architecture/google-cloud.md
@ -0,0 +1,11 @@
+# Google Cloud
+
+## IPv6
+
+Status of IPv6 on Typhoon Google Cloud clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | No        |
+| Node Outbound IPv6      | No        |
+| Kubernetes Ingress IPv6 | Yes       |
--- a/docs/atomic/aws.md
+++ b/docs/atomic/aws.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on AWS with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on AWS with Fedora Atomic.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets. Instances are provisioned on first boot with cloud-init.

@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `aws/fedora-atomic/kubernetes`.

 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.13.1"

  providers = {
    aws = "aws.default"
@ -155,10 +155,10 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.12.1
-ip-10-0-19-112   Ready     34m       v1.12.1
-ip-10-0-4-22     Ready     34m       v1.12.1
+NAME           STATUS  ROLES              AGE  VERSION
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.1
+ip-10-0-26-65  Ready   node               10m  v1.13.1
+ip-10-0-41-21  Ready   node               10m  v1.13.1
 ```

 List the pods.
@ -170,6 +170,7 @@ kube-system   calico-node-1m5bf                         2/2    Running   0
 kube-system   calico-node-7jmr1                         2/2    Running   0         34m              
 kube-system   calico-node-bknc8                         2/2    Running   0         34m              
 kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m              
+kube-system   coredns-1187388186-qjnvp                  1/1    Running   0         34m
 kube-system   kube-apiserver-4mjbk                      1/1    Running   0         34m              
 kube-system   kube-controller-manager-3597210155-j2jbt  1/1    Running   1         34m              
 kube-system   kube-controller-manager-3597210155-j7g7x  1/1    Running   0         34m              
@ -179,7 +180,7 @@ kube-system   kube-proxy-sbbsh                          1/1    Running   0
 kube-system   kube-scheduler-3359497473-5plhf           1/1    Running   0         34m              
 kube-system   kube-scheduler-3359497473-r7zg7           1/1    Running   1         34m              
 kube-system   pod-checkpointer-4kxtl                    1/1    Running   0         34m              
-kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0         33m
+kube-system   pod-checkpointer-4kxtl-ip-10-0-3-155      1/1    Running   0         33m
 ```

 ## Going Further
@ -227,6 +228,7 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | worker_type | EC2 instance type for workers | "t2.small" | See below |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| disk_iops | IOPS of the EBS volume | "0" (i.e. auto) | "400" |
 | worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
--- a/docs/atomic/bare-metal.md
+++ b/docs/atomic/bare-metal.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.

-In this tutorial, we'll network boot and provision a Kubernetes v1.12.1 cluster on bare-metal with Fedora Atomic.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.1 cluster on bare-metal with Fedora Atomic.

 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora Atomic via kickstart, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via cloud-init.

@ -235,7 +235,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-atomic/kubernete

 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.13.1"
  
  providers = {
    local = "local.default"
@ -360,10 +360,10 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
-NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.12.1
-node2.example.com   Ready     11m       v1.12.1
-node3.example.com   Ready     11m       v1.12.1
+NAME                STATUS  ROLES              AGE  VERSION
+node1.example.com   Ready   controller,master  10m  v1.13.1
+node2.example.com   Ready   node               10m  v1.13.1
+node3.example.com   Ready   node               10m  v1.13.1
 ```

 List the pods.
@ -374,6 +374,7 @@ NAMESPACE     NAME                                       READY     STATUS    RES
 kube-system   calico-node-6qp7f                          2/2       Running   1          11m
 kube-system   calico-node-gnjrm                          2/2       Running   0          11m
 kube-system   calico-node-llbgt                          2/2       Running   0          11m
+kube-system   coredns-1187388186-dj3pd                   1/1       Running   0          11m
 kube-system   coredns-1187388186-mx9rt                   1/1       Running   0          11m
 kube-system   kube-apiserver-7336w                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-b9chx   1/1       Running   0          11m
--- a/docs/atomic/digital-ocean.md
+++ b/docs/atomic/digital-ocean.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on DigitalOcean with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on DigitalOcean with Fedora Atomic.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets. Instances are provisioned on first boot with cloud-init.

@ -77,7 +77,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-atomic/kubern

 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.13.1"
  
  providers = {
    digitalocean = "digitalocean.default"
@ -151,10 +151,10 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.12.1
-10.132.115.81    Ready     10m       v1.12.1
-10.132.124.107   Ready     10m       v1.12.1
+NAME               STATUS  ROLES              AGE  VERSION
+nemo-controller-0  Ready   controller,master  10m  v1.13.1
+nemo-worker-0      Ready   node               10m  v1.13.1
+nemo-worker-1      Ready   node               10m  v1.13.1
 ```

 List the pods.
@ -162,19 +162,20 @@ List the pods.
 ```
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
+kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
+kube-system   flannel-1cq1v                              2/2       Running   0          11m
+kube-system   flannel-hq9t0                              2/2       Running   1          11m
+kube-system   flannel-v0g9w                              2/2       Running   0          11m
 kube-system   kube-apiserver-n10qr                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-37gtw   1/1       Running   1          11m
 kube-system   kube-controller-manager-3271970485-p52t5   1/1       Running   0          11m
-kube-system   kube-flannel-1cq1v                         2/2       Running   0          11m
-kube-system   kube-flannel-hq9t0                         2/2       Running   1          11m
-kube-system   kube-flannel-v0g9w                         2/2       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m
 kube-system   kube-proxy-fh3td                           1/1       Running   0          11m
 kube-system   kube-proxy-k35rc                           1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-2bc4c            1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-b7q47            1/1       Running   1          11m
 kube-system   pod-checkpointer-pr1lq                     1/1       Running   0          11m
-kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0          10m
+kube-system   pod-checkpointer-pr1lq-nemo-controller-0   1/1       Running   0          10m
 ```

 ## Going Further
--- a/docs/atomic/google-cloud.md
+++ b/docs/atomic/google-cloud.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Fedora Atomic is alpha. Fedora does not publish official images for Google Cloud so you must prepare them yourself. Expect rough edges and changes.

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on Google Compute Engine with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on Google Compute Engine with Fedora Atomic.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.

@ -121,7 +121,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-atomic/kuberne

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.13.1"
  
  providers = {
    google   = "google.default"
@ -196,10 +196,10 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.1
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.1
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.1
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.1
 ```

 List the pods.
@ -210,6 +210,7 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
--- a/docs/cl/aws.md
+++ b/docs/cl/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on AWS with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -18,23 +18,15 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys

 ```sh
 $ terraform version
-Terraform v0.11.7
+Terraform v0.11.11
 ```

-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```

 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@ -64,6 +56,10 @@ provider "aws" {
  shared_credentials_file = "/home/user/.config/aws/credentials"
 }

+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
  version = "~> 1.0"
  alias = "default"
@ -96,7 +92,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.13.1"

  providers = {
    aws = "aws.default"
@ -168,10 +164,10 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.12.1
-ip-10-0-19-112   Ready     34m       v1.12.1
-ip-10-0-4-22     Ready     34m       v1.12.1
+NAME           STATUS  ROLES              AGE  VERSION
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.1
+ip-10-0-26-65  Ready   node               10m  v1.13.1
+ip-10-0-41-21  Ready   node               10m  v1.13.1
 ```

 List the pods.
@ -183,6 +179,7 @@ kube-system   calico-node-1m5bf                         2/2    Running   0
 kube-system   calico-node-7jmr1                         2/2    Running   0         34m              
 kube-system   calico-node-bknc8                         2/2    Running   0         34m              
 kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m              
+kube-system   coredns-1187388186-qjnvp                  1/1    Running   0         34m
 kube-system   kube-apiserver-4mjbk                      1/1    Running   0         34m              
 kube-system   kube-controller-manager-3597210155-j2jbt  1/1    Running   1         34m              
 kube-system   kube-controller-manager-3597210155-j7g7x  1/1    Running   0         34m              
@ -192,7 +189,7 @@ kube-system   kube-proxy-sbbsh                          1/1    Running   0
 kube-system   kube-scheduler-3359497473-5plhf           1/1    Running   0         34m              
 kube-system   kube-scheduler-3359497473-r7zg7           1/1    Running   1         34m              
 kube-system   pod-checkpointer-4kxtl                    1/1    Running   0         34m              
-kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0         33m
+kube-system   pod-checkpointer-4kxtl-ip-10-0-3-155      1/1    Running   0         33m
 ```

 ## Going Further
@ -244,6 +241,7 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| disk_iops | IOPS of the EBS volume | "0" (i.e. auto) | "400" |
 | worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
 | controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
@ -261,3 +259,8 @@ Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-typ

 !!! tip "MTU"
    If your EC2 instance type supports [Jumbo frames](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) (most do), we recommend you change the `network_mtu` to 8981! You will get better pod-to-pod bandwidth.
+
+#### Spot
+
+Add `worker_price = "0.10"` to use spot instance workers (instead of "on-demand") and set a maximum spot price in USD. Clusters can tolerate spot market interuptions fairly well (reschedules pods, but cannot drain) to save money, with the tradeoff that requests for workers may go unfulfilled.
+
--- a/docs/cl/azure.md
+++ b/docs/cl/azure.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on Azure with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on Azure with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -21,23 +21,15 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys

 ```sh
 $ terraform version
-Terraform v0.11.7
+Terraform v0.11.11
 ```

-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```

 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@ -62,6 +54,10 @@ provider "azurerm" {
  alias   = "default"
 }

+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
  version = "~> 1.0"
  alias   = "default"
@ -91,7 +87,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.

 ```tf
 module "azure-ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.13.1"

  providers = {
    azurerm  = "azurerm.default"
@ -112,7 +108,7 @@ module "azure-ramius" {
  asset_dir          = "/home/user/.secrets/clusters/ramius"

  # optional
-  worker_count    = 3
+  worker_count    = 2
  host_cidr       = "10.0.0.0/20"
 }
 ```
@ -165,10 +161,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/ramius/auth/kubeconfig
 $ kubectl get nodes
 NAME                  STATUS  ROLES              AGE  VERSION
-ramius-controller-0   Ready   controller,master  24m  v1.12.1
-ramius-worker-000001  Ready   node               25m  v1.12.1
-ramius-worker-000002  Ready   node               24m  v1.12.1
-ramius-worker-000005  Ready   node               24m  v1.12.1
+ramius-controller-0   Ready   controller,master  24m  v1.13.1
+ramius-worker-000001  Ready   node               25m  v1.13.1
+ramius-worker-000002  Ready   node               24m  v1.13.1
 ```

 List the pods.
@ -177,17 +172,16 @@ List the pods.
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                        READY  STATUS    RESTARTS  AGE
 kube-system   coredns-7c6fbb4f4b-b6qzx                    1/1    Running   0         26m
+kube-system   coredns-7c6fbb4f4b-j2k3d                    1/1    Running   0         26m
+kube-system   flannel-bwf24                               2/2    Running   2         26m
+kube-system   flannel-ks5qb                               2/2    Running   0         26m
+kube-system   flannel-tq2wg                               2/2    Running   0         26m
 kube-system   kube-apiserver-hxgsx                        1/1    Running   3         26m
 kube-system   kube-controller-manager-5ff9cd7bb6-b942n    1/1    Running   0         26m
 kube-system   kube-controller-manager-5ff9cd7bb6-bbr6w    1/1    Running   0         26m
-kube-system   kube-flannel-bwf24                          2/2    Running   2         26m
-kube-system   kube-flannel-ks5qb                          2/2    Running   0         26m
-kube-system   kube-flannel-nghsx                          2/2    Running   2         26m
-kube-system   kube-flannel-tq2wg                          2/2    Running   0         26m
 kube-system   kube-proxy-j4vpq                            1/1    Running   0         26m
 kube-system   kube-proxy-jxr5d                            1/1    Running   0         26m
 kube-system   kube-proxy-lbdw5                            1/1    Running   0         26m
-kube-system   kube-proxy-v8r7c                            1/1    Running   0         26m
 kube-system   kube-scheduler-5f76d69686-s4fbx             1/1    Running   0         26m
 kube-system   kube-scheduler-5f76d69686-vgdgn             1/1    Running   0         26m
 kube-system   pod-checkpointer-cnqdg                      1/1    Running   0         26m
--- a/docs/cl/bare-metal.md
+++ b/docs/cl/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.12.1 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.1 cluster on bare-metal with Container Linux.

 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -113,28 +113,20 @@ $ terraform version
 Terraform v0.11.7
 ```

-Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system.
+Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
 wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
 tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-sudo mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox /usr/local/bin/
+mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.2
 ```

-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  matchbox = "/usr/local/bin/terraform-provider-matchbox"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```

 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@ -149,12 +141,17 @@ Configure the Matchbox provider to use your Matchbox API endpoint and client cer

 ```tf
 provider "matchbox" {
+  version = "0.2.2"
  endpoint    = "matchbox.example.com:8081"
  client_cert = "${file("~/.config/matchbox/client.crt")}"
  client_key  = "${file("~/.config/matchbox/client.key")}"
  ca          = "${file("~/.config/matchbox/ca.crt")}"
 }

+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
  version = "~> 1.0"
  alias = "default"
@ -182,7 +179,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.1"
  
  providers = {
    local = "local.default"
@ -291,9 +288,9 @@ Apply complete! Resources: 55 added, 0 changed, 0 destroyed.
 To watch the install to disk (until machines reboot from disk), SSH to port 2222.

 ```
-# before v1.12.1
+# before v1.13.1
 $ ssh debug@node1.example.com
-# after v1.12.1
+# after v1.13.1
 $ ssh -p 2222 core@node1.example.com
 ```

@ -317,10 +314,10 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
-NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.12.1
-node2.example.com   Ready     11m       v1.12.1
-node3.example.com   Ready     11m       v1.12.1
+NAME                STATUS  ROLES              AGE  VERSION
+node1.example.com   Ready   controller,master  10m  v1.13.1
+node2.example.com   Ready   node               10m  v1.13.1
+node3.example.com   Ready   node               10m  v1.13.1
 ```

 List the pods.
@ -331,6 +328,7 @@ NAMESPACE     NAME                                       READY     STATUS    RES
 kube-system   calico-node-6qp7f                          2/2       Running   1          11m
 kube-system   calico-node-gnjrm                          2/2       Running   0          11m
 kube-system   calico-node-llbgt                          2/2       Running   0          11m
+kube-system   coredns-1187388186-dj3pd                   1/1       Running   0          11m
 kube-system   coredns-1187388186-mx9rt                   1/1       Running   0          11m
 kube-system   kube-apiserver-7336w                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-b9chx   1/1       Running   0          11m
--- a/docs/cl/digital-ocean.md
+++ b/docs/cl/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on DigitalOcean with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -18,23 +18,15 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys

 ```sh
 $ terraform version
-Terraform v0.11.7
+Terraform v0.11.11
 ```

-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```

 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@ -63,6 +55,10 @@ provider "digitalocean" {
  alias = "default"
 }

+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
  version = "~> 1.0"
  alias = "default"
@ -90,7 +86,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube

 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.13.1"
  
  providers = {
    digitalocean = "digitalocean.default"
@ -163,10 +159,10 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.12.1
-10.132.115.81    Ready     10m       v1.12.1
-10.132.124.107   Ready     10m       v1.12.1
+NAME               STATUS  ROLES              AGE  VERSION
+nemo-controller-0  Ready   controller,master  10m  v1.13.1
+nemo-worker-0      Ready   node               10m  v1.13.1
+nemo-worker-1      Ready   node               10m  v1.13.1
 ```

 List the pods.
@ -174,19 +170,20 @@ List the pods.
 ```
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
+kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
+kube-system   flannel-1cq1v                              2/2       Running   0          11m
+kube-system   flannel-hq9t0                              2/2       Running   1          11m
+kube-system   flannel-v0g9w                              2/2       Running   0          11m
 kube-system   kube-apiserver-n10qr                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-37gtw   1/1       Running   1          11m
 kube-system   kube-controller-manager-3271970485-p52t5   1/1       Running   0          11m
-kube-system   kube-flannel-1cq1v                         2/2       Running   0          11m
-kube-system   kube-flannel-hq9t0                         2/2       Running   1          11m
-kube-system   kube-flannel-v0g9w                         2/2       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m
 kube-system   kube-proxy-fh3td                           1/1       Running   0          11m
 kube-system   kube-proxy-k35rc                           1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-2bc4c            1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-b7q47            1/1       Running   1          11m
 kube-system   pod-checkpointer-pr1lq                     1/1       Running   0          11m
-kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0          10m
+kube-system   pod-checkpointer-pr1lq-nemo-controller-0   1/1       Running   0          10m
 ```

 ## Going Further
--- a/docs/cl/google-cloud.md
+++ b/docs/cl/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.12.1 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.1 cluster on Google Compute Engine with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -21,20 +21,12 @@ $ terraform version
 Terraform v0.11.7
 ```

-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.

 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```

 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@ -65,6 +57,10 @@ provider "google" {
  region      = "us-central1"
 }

+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
  version = "~> 1.0"
  alias = "default"
@ -97,7 +93,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.1"
  
  providers = {
    google   = "google.default"
@ -171,10 +167,10 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.1
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.1
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.1
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.1
 ```

 List the pods.
@ -185,6 +181,7 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
--- a/docs/index.md
+++ b/docs/index.md
@ -11,29 +11,32 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)

 ## Modules

-Typhoon provides a Terraform Module for each supported operating system and platform.
+Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](atomic/aws.md) | alpha |
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](atomic/bare-metal.md) | alpha |
-| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](atomic/digital-ocean.md) | alpha |
-| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |
-| Google Cloud  | Fedora Atomic    | [google-cloud/container-linux/kubernetes](atomic/google-cloud.md) | alpha |
+| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
+| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |

-The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
+Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
+| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
+| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |

 ## Documentation

@ -46,7 +49,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.1"
  
  providers = {
    google   = "google.default"
@ -86,10 +89,10 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.1
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.1
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.1
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.1
 ```

 List the pods.
@ -100,6 +103,7 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
@ -110,6 +114,7 @@ kube-system   kube-proxy-njn47                          1/1    Running   0
 kube-system   kube-scheduler-3895335239-5x87r           1/1    Running   0         6m
 kube-system   kube-scheduler-3895335239-bzrrt           1/1    Running   1         6m
 kube-system   pod-checkpointer-l6lrt                    1/1    Running   0         6m
+kube-system   pod-checkpointer-l6lrt-controller-0       1/1    Running   0         6m
 ```

 ## Help
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -18,7 +18,7 @@ module "google-cloud-yavin" {
 }

 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.1"
  ...
 }
 ```
@ -126,3 +126,147 @@ Typhoon supports multi-controller clusters, so it is possible to upgrade a clust

 !!! warning
    Typhoon does not support or document node replacement as an upgrade strategy. It limits Typhoon's ability to make infrastructure and architectural changes between tagged releases. 
+
+### Terraform Plugins Directory
+
+Use the Terraform 3rd-party [plugin directory](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) `~/.terraform.d/plugins` to keep versioned copies of the `terraform-provider-ct` and `terraform-provider-matchbox` plugins. The plugin directory replaces the `~/.terraformrc` file to allow 3rd party plugins to be defined and versioned independently (rather than globally).
+
+```
+# ~/.terraformrc (DEPRECATED)
+providers {
+  ct = "/usr/local/bin/terraform-provider-ct"
+  matchbox = "/usr/local/bin/terraform-provider-matchbox"
+}
+```
+
+Migrate to using the Terraform plugin directory. Move `~/.terraformrc` to a backup location.
+
+```
+mv ~/.terraformrc ~/.terraform-backup
+```
+
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`. Download the **same version** of `terraform-provider-ct` you were using with `~/.terraformrc`, updating only be done as a followup and is **only** safe for v1.12.2+ clusters!
+
+```sh
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.2.1
+```
+
+If you use bare-metal, add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the versioned name.
+
+```sh
+wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
+tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
+mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.2
+```
+
+Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
+
+```
+$ tree ~/.terraform.d/
+/home/user/.terraform.d/
+└── plugins
+    ├── terraform-provider-ct_v0.2.1
+    └── terraform-provider-matchbox_v0.2.2
+```
+
+In each Terraform working directory, set the version of each provider.
+
+```
+# providers.tf
+
+provider "matchbox" {
+  version = "0.2.2"
+  ...
+}
+
+provider "ct" {
+  version = "0.2.1"
+}
+```
+
+Run `terraform init` to ensure plugin version requirements are met. Verify `terraform plan` does not produce a diff, since the plugin versions should be the same as previously.
+
+```
+$ terraform init
+$ terraform plan
+```
+
+### Upgrade terraform-provider-ct
+
+The [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin parses, validates, and converts Container Linux Configs into Ignition user-data for provisioning instances. Previously, updating the plugin re-provisioned controller nodes and was destructive to clusters. With Typhoon v1.12.2+, the plugin can be updated in-place and on apply, only workers will be replaced.
+
+First, [migrate](#terraform-plugins-directory) to the Terraform 3rd-party plugin directory to allow 3rd-party plugins to be defined and versioned independently (rather than globally).
+
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
+
+```sh
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
+```
+
+Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
+
+```
+$ tree ~/.terraform.d/
+/home/user/.terraform.d/
+└── plugins
+    ├── terraform-provider-ct_v0.2.1
+    ├── terraform-provider-ct_v0.3.0
+    └── terraform-provider-matchbox_v0.2.2
+```
+
+
+Update the version of the `ct` plugin in each Terraform working directory. Typhoon clusters managed in the working directory **must** be v1.12.2 or higher.
+
+```
+# providers.tf
+provider "ct" {
+  version = "0.3.0"
+}
+```
+
+Run init and plan to check that no diff is proposed for the controller nodes (a diff would destroy cluster state).
+
+```
+terraform init
+terraform plan
+```
+
+Apply the change. Worker nodes' user-data will be changed and workers will be replaced. Rollout happens slightly differently on each platform:
+
+
+#### AWS
+
+AWS creates a new worker ASG, then removes the old ASG. New workers join the cluster and old workers disappear. `terraform apply` will hang during this process.
+
+#### Azure
+
+Azure edits the worker scale set in-place instantly. Manually terminate workers to create replacement workers using the new user-data.
+
+#### Bare-Metal
+
+No action is needed. Bare-Metal machines do not re-PXE unless explicitly made to do so.
+
+#### DigitalOcean
+
+DigitalOcean destroys existing worker nodes and DNS records, then creates new workers and DNS records. DigitalOcean lacks a "managed group" notion. For worker droplets to join the cluster, you **must** taint the secret copying step to indicate it must be repeated to add the kubeconfig to new workers.
+
+```
+# old workers destroyed, new workers created
+terraform apply
+
+# add kubeconfig to new workers
+terraform state list | grep null_resource
+terraform taint -module digital-ocean-nemo null_resource.copy-worker-secrets.N
+terraform apply
+```
+
+Expect downtime.
+
+#### Google Cloud
+
+Google Cloud creates a new worker template and edits the worker instance group instantly. Manually terminate workers and replacement workers will use the user-data.
+
--- a/docs/topics/performance.md
+++ b/docs/topics/performance.md
@ -24,17 +24,17 @@ Network performance varies based on the platform and CNI plugin. `iperf` was use

 | Platform / Plugin          | Theory | Host to Host | Pod to Pod   |
 |----------------------------|-------:|-------------:|-------------:|
-| AWS (flannel)              | ?      | 976 MB/s     | 900-999 MB/s |
-| AWS (calico, MTU 1480)     | ?      | 976 MB/s     | 100-350 MB/s |
-| AWS (calico, MTU 8981)     | ?      | 976 MB/s     | 900-999 MB/s |
-| Azure (flannel)            | ?      | 749 MB/s     | 680 MB/s     |
-| Bare-Metal (flannel)       | 1 GB/s | ~940 MB/s    | 903 MB/s     | 
-| Bare-Metal (calico)        | 1 GB/s | ~940 MB/s    | 931 MB/s     |
-| Bare-Metal (flannel, bond) | 3 GB/s |  2.3 GB/s    | 1.17 GB/s    | 
-| Bare-Metal (calico, bond)  | 3 GB/s |  2.3 GB/s    | 1.17 GB/s    |
-| Digital Ocean              | ?      | ~940 MB/s     | 820-880 MB/s |
-| Google Cloud (flannel)     | ?      | 1.94 GB/s    | 1.76 GB/s    |
-| Google Cloud (calico)      | ?      | 1.94 GB/s    | 1.81 GB/s    |
+| AWS (flannel)              | Varies | 976 Mb/s     | 900-999 Mb/s |
+| AWS (calico, MTU 1480)     | Varies | 976 Mb/s     | 100-350 Mb/s |
+| AWS (calico, MTU 8981)     | Varies | 976 Mb/s     | 900-999 Mb/s |
+| Azure (flannel)            | Varies | 749 Mb/s     | 680 Mb/s     |
+| Bare-Metal (flannel)       | 1 Gb/s | ~940 Mb/s    | 903 Mb/s     |
+| Bare-Metal (calico)        | 1 Gb/s | ~940 Mb/s    | 931 Mb/s     |
+| Bare-Metal (flannel, bond) | 3 Gb/s |  2.3 Gb/s    | 1.17 Gb/s    |
+| Bare-Metal (calico, bond)  | 3 Gb/s |  2.3 Gb/s    | 1.17 Gb/s    |
+| Digital Ocean              | 2 Gb/s | 1.97 Gb/s    | 1.64 Gb/s    |
+| Google Cloud (flannel)     | 2 Gb/s | 1.94 Gb/s    | 1.76 Gb/s    |
+| Google Cloud (calico)      | 2 Gb/s | 1.94 Gb/s    | 1.81 Gb/s    |

 Notes:

--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/google-cloud/container-linux/kubernetes/apiserver.tf
+++ b/google-cloud/container-linux/kubernetes/apiserver.tf
@ -42,7 +42,7 @@ resource "google_compute_backend_service" "apiserver" {
  protocol         = "TCP"
  port_name        = "apiserver"
  session_affinity = "NONE"
-  timeout_sec      = "60"
+  timeout_sec      = "300"

  # controller(s) spread across zonal instance groups
  backend {
--- a/google-cloud/container-linux/kubernetes/bootkube.tf
+++ b/google-cloud/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=d14348a368298b7c8b0878accba4974cce5401f9"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@ -11,6 +11,7 @@ module "bootkube" {
  pod_cidr              = "${var.pod_cidr}"
  service_cidr          = "${var.service_cidr}"
  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"

  // temporary
  apiserver_port = 443
--- a/google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -90,6 +90,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -123,7 +124,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.1
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -143,17 +144,14 @@ storage:
          set -e
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
            --mount volume=assets,target=/assets \
            --volume bootstrap,kind=host,source=/etc/kubernetes \
            --mount volume=bootstrap,target=/etc/kubernetes \
            $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
            --net=host \
            --dns=host \
            --exec=/bootkube -- start --asset-dir=/assets "$@"
--- a/google-cloud/container-linux/kubernetes/controllers.tf
+++ b/google-cloud/container-linux/kubernetes/controllers.tf
@ -57,6 +57,12 @@ resource "google_compute_instance" "controllers" {

  can_ip_forward = true
  tags           = ["${var.cluster_name}-controller"]
+
+  lifecycle {
+    ignore_changes = [
+      "metadata",
+    ]
+  }
 }

 # Controller Ignition configs
--- a/google-cloud/container-linux/kubernetes/ingress.tf
+++ b/google-cloud/container-linux/kubernetes/ingress.tf
@ -1,13 +1,19 @@
-# Static IPv4 address for the TCP Proxy Load Balancer
+# Static IPv4 address for Ingress Load Balancing
 resource "google_compute_global_address" "ingress-ipv4" {
-  name       = "${var.cluster_name}-ingress-ip"
+  name       = "${var.cluster_name}-ingress-ipv4"
  ip_version = "IPV4"
 }

+# Static IPv6 address for Ingress Load Balancing
+resource "google_compute_global_address" "ingress-ipv6" {
+  name       = "${var.cluster_name}-ingress-ipv6"
+  ip_version = "IPV6"
+}
+
 # Forward IPv4 TCP traffic to the HTTP proxy load balancer
 # Google Cloud does not allow TCP proxies for port 80. Must use HTTP proxy.
-resource "google_compute_global_forwarding_rule" "ingress-http" {
-  name        = "${var.cluster_name}-ingress-http"
+resource "google_compute_global_forwarding_rule" "ingress-http-ipv4" {
+  name        = "${var.cluster_name}-ingress-http-ipv4"
  ip_address  = "${google_compute_global_address.ingress-ipv4.address}"
  ip_protocol = "TCP"
  port_range  = "80"
@ -15,14 +21,33 @@ resource "google_compute_global_forwarding_rule" "ingress-http" {
 }

 # Forward IPv4 TCP traffic to the TCP proxy load balancer
-resource "google_compute_global_forwarding_rule" "ingress-https" {
-  name        = "${var.cluster_name}-ingress-https"
+resource "google_compute_global_forwarding_rule" "ingress-https-ipv4" {
+  name        = "${var.cluster_name}-ingress-https-ipv4"
  ip_address  = "${google_compute_global_address.ingress-ipv4.address}"
  ip_protocol = "TCP"
  port_range  = "443"
  target      = "${google_compute_target_tcp_proxy.ingress-https.self_link}"
 }

+# Forward IPv6 TCP traffic to the HTTP proxy load balancer
+# Google Cloud does not allow TCP proxies for port 80. Must use HTTP proxy.
+resource "google_compute_global_forwarding_rule" "ingress-http-ipv6" {
+  name        = "${var.cluster_name}-ingress-http-ipv6"
+  ip_address  = "${google_compute_global_address.ingress-ipv6.address}"
+  ip_protocol = "TCP"
+  port_range  = "80"
+  target      = "${google_compute_target_http_proxy.ingress-http.self_link}"
+}
+
+# Forward IPv6 TCP traffic to the TCP proxy load balancer
+resource "google_compute_global_forwarding_rule" "ingress-https-ipv6" {
+  name        = "${var.cluster_name}-ingress-https-ipv6"
+  ip_address  = "${google_compute_global_address.ingress-ipv6.address}"
+  ip_protocol = "TCP"
+  port_range  = "443"
+  target      = "${google_compute_target_tcp_proxy.ingress-https.self_link}"
+}
+
 # HTTP proxy load balancer for ingress controllers
 resource "google_compute_target_http_proxy" "ingress-http" {
  name        = "${var.cluster_name}-ingress-http"
--- a/google-cloud/container-linux/kubernetes/network.tf
+++ b/google-cloud/container-linux/kubernetes/network.tf
@ -57,12 +57,12 @@ resource "google_compute_firewall" "allow-apiserver" {
  target_tags   = ["${var.cluster_name}-controller"]
 }

-# Calico BGP and IPIP
-# https://docs.projectcalico.org/v2.5/reference/public-cloud/gce
-resource "google_compute_firewall" "internal-calico" {
-  count = "${var.networking == "calico" ? 1 : 0}"
+# BGP and IPIP
+# https://docs.projectcalico.org/latest/reference/public-cloud/gce
+resource "google_compute_firewall" "internal-bgp" {
+  count = "${var.networking != "flannel" ? 1 : 0}"

-  name    = "${var.cluster_name}-internal-calico"
+  name    = "${var.cluster_name}-internal-bgp"
  network = "${google_compute_network.network.name}"

  allow {
@ -123,20 +123,6 @@ resource "google_compute_firewall" "internal-kubelet" {
  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }

-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "google_compute_firewall" "internal-kubelet-readonly" {
-  name    = "${var.cluster_name}-internal-kubelet-readonly"
-  network = "${google_compute_network.network.name}"
-
-  allow {
-    protocol = "tcp"
-    ports    = [10255]
-  }
-
-  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
-  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
-}
-
 # Workers

 resource "google_compute_firewall" "allow-ingress" {
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Dalton Hubble	4d2f33aee6	Update changelog for v1.13.1 release	2018-12-17 14:28:27 -08:00
Dalton Hubble	d42f47c49e	Update terraform-provider-ct plugin from v0.2.1 to v0.3.0 * Provide migration instructions for upgrading terraform-provider-ct in-place for v1.12.2+ clusters * Require switching from ~/.terraformrc to the Terraform third-party plugins directory ~/.terraform.d/plugins/ * Require Container Linux 1688.5.3 or newer	2018-12-17 14:13:50 -08:00
Dalton Hubble	53e549f233	Add Flatcar Linux to the issue template	2018-12-16 10:47:59 -08:00
Dalton Hubble	bcb200186d	Add admin kubeconfig as a Terraform output * May be used to write a local file	2018-12-15 22:52:28 -08:00
Dalton Hubble	479d498024	Update Calico from v3.3.2 to v3.4.0 * https://docs.projectcalico.org/v3.4/releases/	2018-12-15 18:05:16 -08:00
Dalton Hubble	e0c032be94	Increase GCP TCP proxy apiserver backend timeout to 5 minutes * On GCP, kubectl port-forward connections to pods are closed after a timeout (unlike AWS NLB's or Azure load balancers) * Increase the GCP apiserver backend service timeout from 1 minute to 5 minutes to be more similar to AWS/Azure LB behavior	2018-12-15 17:34:18 -08:00
Dalton Hubble	b74bf11772	Update Grafana from v5.4.0 to v5.4.2 * https://github.com/grafana/grafana/releases/tag/v5.4.2 * https://github.com/grafana/grafana/releases/tag/v5.4.1	2018-12-15 12:39:03 -08:00
Dalton Hubble	018c5edc25	Update Kubernetes from v1.13.0 to v1.13.1 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1131	2018-12-15 11:44:57 -08:00
Lucas Servén Marín	8aeec0b9b5	Fix typo in descriptive firewall name (#359 )	2018-12-15 11:34:32 -08:00
Dalton Hubble	ff6ab571f3	Update Calico from v3.3.1 to v3.3.2 * https://docs.projectcalico.org/v3.3/releases/	2018-12-06 22:56:55 -08:00
Dalton Hubble	991fb44c37	Update Grafana from v5.3.4 to v5.4.0 * https://github.com/grafana/grafana/releases/tag/v5.4.0	2018-12-06 01:33:50 -08:00
Dalton Hubble	d31f444fcd	Update Kubernetes from v1.12.3 to v1.13.0	2018-12-03 20:44:32 -08:00
Dalton Hubble	76d993cdae	Add experimental kube-router CNI provider * Add kube-router for pod networking and NetworkPolicy as an experiment * Experiments are not documented or supported in any way, and may be removed without notice. They have known issues and aren't enabled without special options.	2018-12-03 19:52:28 -08:00
Dalton Hubble	b6016d0a26	Disable Grafana login form, admin user can't be disabled * Example manifests aim to provide a read-only dashboard visible to any users with network access (i.e. kubectl port-forward, LAN) * Problem: Grafana always has an admin user, even with the user management system disabled * Disable the login form to prevent admin login	2018-11-28 22:04:08 -08:00
Dalton Hubble	eec314b52f	Update CHANGES changelog for release	2018-11-28 09:23:13 -08:00
yokhahn	bcce02a9ce	Add Kubelet /etc/iscsi and iscsiadm mounts on bare-metal * Allow using iSCSI with Container Linux bare-metal clusters * Warning, iSCSI isn't part of Kubernetes conformance and isn't regularly evaluated	2018-11-28 00:28:46 -08:00
Dalton Hubble	42c523e6a2	Recommend switch from ~/.terraformrc to 3rd-party plugin dir * Switch tutorials from using ~/.terraformrc to using the 3rd-party plugin directory so 3rd-party plugins can be pinned * Continue to show using terraform-provider-ct v0.2.2. Updating to a newer version is only safe once all managed clusters are v1.12.2 or higher	2018-11-28 00:03:15 -08:00
Dalton Hubble	64b4c10418	Improve features and modules list docs * Remove bullet about isolating workloads on workers, its now common practice and new users will assume it * List advanced features available in each module * Fix erroneous Kubernetes version listing for Google Cloud Fedora Atomic	2018-11-26 22:58:00 -08:00
Dalton Hubble	872b11b948	Update ngninx-ingress from v0.20.0 to v0.21.0 * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.21.0	2018-11-26 21:57:34 -08:00
Dalton Hubble	5b27d8d889	Update Kubernetes from v1.12.2 to v1.12.3 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md/#v1123	2018-11-26 21:06:09 -08:00
Dalton Hubble	840b73f9ba	Update pod-checkpointer image to query Kubelet secure API * Updates pod-checkpointer to prefer the Kubelet secure API (before falling back to the Kubelet read-only API that is disabled on Typhoon clusters since https://github.com/poseidon/typhoon/pull/324) * Previously, pod-checkpointer checkpointed an initial set of pods during bootstrapping so recovery from power cycling clusters was unaffected, but logs were noisy * https://github.com/kubernetes-incubator/bootkube/pull/1027 * https://github.com/kubernetes-incubator/bootkube/pull/1025	2018-11-26 20:24:32 -08:00
Dalton Hubble	915af3c6cc	Fix Calico Felix reporting usage data, require opt-in * Calico Felix has been reporting anonymous usage data about the version and cluster size, which violates Typhoon's privacy policy where analytics should be opt-in only * Add a variable enable_reporting (default: false) to allow opting in to reporting usage data to Calico (or future components)	2018-11-20 01:03:00 -08:00
Dalton Hubble	c6586b69fd	Use eviction policy Delete for Low priority VMSS workers * Fix issue where Azure defaults to Deallocate eviction policy, which required manually restarting deallocated workers * Require terraform-provider-azurerm v1.19+ to support setting the eviction_policy	2018-11-18 21:04:50 -08:00
Dalton Hubble	ea3fc6d2a7	Update CoreDNS from v1.2.4 to v1.2.6 * https://coredns.io/2018/11/05/coredns-1.2.6-release/	2018-11-18 16:45:53 -08:00
Dalton Hubble	c8c43f3991	Update Grafana from v5.3.2 to v5.3.4 * https://github.com/grafana/grafana/releases/tag/v5.3.3 * https://github.com/grafana/grafana/releases/tag/v5.3.4	2018-11-18 16:42:50 -08:00
Dalton Hubble	58472438ce	Update mkdocs-material theme from v3.0.6 to v3.1.0 * https://github.com/squidfunk/mkdocs-material/releases/tag/3.1.0	2018-11-18 16:08:00 -08:00
Dalton Hubble	7f8e781ae4	Measure DigitalOcean network performance * Measuring pod-to-pod bandwidth in a few regions (NYC3, FRA1, SFO1) shows DigitalOcean has made some improvements	2018-11-11 21:08:10 -08:00
Dalton Hubble	56e9a82984	Add flannel resource request and mount only /run/flannel	2018-11-11 20:35:21 -08:00
Dalton Hubble	e95b856a22	Enable CoreDNS loop and loadbalance plugins * loop sends an initial query to detect infinite forwarding loops in configured upstream DNS servers and fast exit with an error (its a fatal misconfiguration on the network that will otherwise cause resolvers to consume memory/CPU until crashing, masking the problem) * https://github.com/coredns/coredns/tree/master/plugin/loop * loadbalance randomizes the ordering of A, AAAA, and MX records in responses to provide round-robin load balancing (as usual, clients may still cache responses though) * https://github.com/coredns/coredns/tree/master/plugin/loadbalance	2018-11-10 17:36:56 -08:00
Dalton Hubble	31f48a81a8	Update docs to show flannel DaemonSet instead of kube-flannel * No functional change, the rename is just for consistency	2018-11-10 15:16:06 -08:00
Dalton Hubble	2b3f61d1bb	Update Calico from v3.3.0 to v3.3.1 * Structure Calico and flannel manifests * Rename kube-flannel mentions to just flannel	2018-11-10 13:37:12 -08:00
Dalton Hubble	8fd2978c31	Update bootkube image version from v0.13.0 to v0.14.0 * https://github.com/kubernetes-incubator/bootkube/releases/tag/v0.14.0	2018-11-06 23:35:11 -08:00
Dalton Hubble	7de03a1279	Fix Prometheus etcd scrape config for DigitalOcean * Kubelet uses a node's hostname as the node name, which isn't resolvable on DigitalOcean. On DigitalOcean, the node name was set to the internal IP until #337 switched to instead configuring kube-apiserver to prefer the InternalIP for communication * Explicitly configure etcd scrapes to target each controller by internal IP and port 2381 (replace __address__)	2018-11-06 23:02:45 -08:00
Dalton Hubble	be9f7b87d6	Update Prometheus from v2.4.3 to v2.5.0 * https://github.com/prometheus/prometheus/releases/tag/v2.5.0	2018-11-06 22:16:12 -08:00
Dalton Hubble	721c847943	Set kube-apiserver kubelet preferred address types * Prefer InternalIP and ExternalIP over the node's hostname, to match upstream behavior and kubeadm * Previously, hostname-override was used to set node names to internal IP's to work around some cloud providers not resolving hostnames for instances (e.g. DO droplets)	2018-11-03 22:31:55 -07:00
Dalton Hubble	78c9fdc18f	Update mkdocs-material docs theme version	2018-10-28 19:45:58 -07:00
Dalton Hubble	884c8b39dc	Update Grafana from v5.3.1 to v5.3.2 * https://github.com/grafana/grafana/releases/tag/v5.3.2	2018-10-28 19:44:22 -07:00
Dalton Hubble	0e71f7e565	Ignore controller user_data changes to allow plugin updates * Updating the `terraform-provider-ct` plugin is known to produce a `user_data` diff in all pre-existing clusters. Applying the diff to pre-existing cluster destroys controller nodes * Ignore changes to controller `user_data`. Once all managed clusters use a release containing this change, it is possible to update the `terraform-provider-ct` plugin (worker `user_data` will still be modified) * Changing the module `ref` for an existing cluster and re-applying is still NOT supported (although this PR would protect controllers from being destroyed)	2018-10-28 16:48:12 -07:00
Dalton Hubble	8c4200d425	Add DigitalOcean AAAA DNS records on Fedora Atomic	2018-10-28 14:57:31 -07:00
Dalton Hubble	5be5b261e2	Add an IPv6 address and forwarding rules on Google Cloud * Allowing serving IPv6 applications via Kubernetes Ingress on Typhoon Google Cloud clusters * Add `ingress_static_ipv6` output variable for use in AAAA DNS records	2018-10-28 14:30:58 -07:00
Dalton Hubble	f034ef90ae	Add DigitalOcean AAAA DNS records resolving to workers * Improve the workers "round-robin" DNS FQDN that is created with each cluster by adding AAAA records * CNAME's resolving to the DigitalOcean `workers_dns` output can be followed to find a droplet's IPv4 or IPv6 address * The CNI portmap plugin doesn't support IPv6. Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon with `hostNetwork: true`	2018-10-27 23:09:24 -07:00
Dalton Hubble	3bba1ba0dc	Use new azurerm_network_interface_backend_address_pool_association * Require terraform-provider-azurerm v1.17+ * Inline load_balancer_backend_address_pools_ids is deprecated and scheduled for removal in the v2.0 provider * https://github.com/terraform-providers/terraform-provider-azurerm/pull/2079	2018-10-27 22:55:05 -07:00
Dalton Hubble	dbe7604b67	Add primary field to ip_configuration required by Azure * Required by terraform-provider-azurerm v1.17+ * https://github.com/terraform-providers/terraform-provider-azurerm/pull/2035	2018-10-27 16:44:44 -07:00
Dalton Hubble	9b405a19b2	Fix minor naming inconsistencies in Ignition and CLC data	2018-10-27 16:24:59 -07:00
Tamer Fahmy	bfa1a679eb	Name AWS and DigitalOcean Ignition data sources consistently	2018-10-27 16:14:44 -07:00
Dalton Hubble	f1da0731d8	Update Kubernetes from v1.12.1 to v1.12.2 * Update CoreDNS from v1.2.2 to v1.2.4 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122 * https://coredns.io/2018/10/17/coredns-1.2.4-release/ * https://coredns.io/2018/10/16/coredns-1.2.3-release/	2018-10-27 15:47:57 -07:00
Dalton Hubble	d641a058fe	Update Calico from v3.2.3 to v3.3.0 * https://docs.projectcalico.org/v3.3/releases/	2018-10-23 20:30:30 -07:00
Dalton Hubble	99a6d5478b	Disable Kubelet read-only port 10255 * We can finally disable the Kubelet read-only port 10255! * Journey: https://github.com/poseidon/typhoon/issues/322#issuecomment-431073073	2018-10-18 21:14:14 -07:00
Dalton Hubble	bc750aec33	Configure Heapster to source metrics from Kubelet authenticated API * Heapster can now get nodes (i.e. kubelets) from the apiserver and source metrics from the Kubelet authenticated API (10250) instead of the Kubelet HTTP read-only API (10255) * https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md * Use the heapster service account token via Kubelet bearer token authn/authz. * Permit Heapster to skip CA verification. The CA cert does not contain IP SANs and cannot since nodes get random IPs that aren't known upfront. Heapster obtains the node list from the apiserver, so the risk of spoofing a node is limited. For the same reason, Prometheus scrapes must skip CA verification for scraping Kubelet's provided by the apiserver. * https://github.com/poseidon/typhoon/blob/v1.12.1/addons/prometheus/config.yaml#L68 * Create a heapster ClusterRole to work around the default Kubernetes `system:heapster` ClusterRole lacking the proper GET `nodes/stats` access. See https://github.com/kubernetes/heapster/issues/1936	2018-10-18 21:03:01 -07:00
Dalton Hubble	d55bfd5589	Fix CoreDNS AntiAffinity spec to prefer spreading replicas * Pods were still being scheduled at random due to a typo	2018-10-17 22:19:57 -07:00
Robert Fairburn	0be4673e44	Add disk_iops variable for AWS * Setting disk_iops is required for disk_type io1 * https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#EBSVolumeTypes	2018-10-17 22:18:54 -07:00
Dalton Hubble	3b44972d78	Add links to header to CHANGES	2018-10-17 09:08:58 -07:00