Add CHANGES note about reducing the pod eviciton timeout

* https://github.com/prometheus/prometheus/releases/tag/v2.7.1

.github/ISSUE_TEMPLATE.md

@@ -5,8 +5,8 @@
 ### Environment
 
 * Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, fedora-atomic
-* Ref: Release version or Git SHA (reporting latest is **not** helpful)
+* OS: container-linux, flatcar-linux, or fedora-atomic
+* Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
 * Terraform: `terraform version` (reporting latest is **not** helpful)
 * Plugins: Provider plugin versions (reporting latest is **not** helpful)
 

CHANGES.md

@@ -4,6 +4,125 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.13.3
+
+* Kubernetes [v1.13.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1133)
+* Update etcd from v3.3.10 to [v3.3.11](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3311-2019-1-11)
+* Update CoreDNS from v1.3.0 to [v1.3.1](https://coredns.io/2019/01/13/coredns-1.3.1-release/)
+  * Switch from the `proxy` plugin to the faster `forward` plugin for upsteam resolvers
+* Update Calico from v3.4.0 to [v3.5.0](https://docs.projectcalico.org/v3.5/releases/)
+* Update flannel from v0.10.0 to [v0.11.0](https://github.com/coreos/flannel/releases/tag/v0.11.0)
+* Reduce pod eviction timeout for deleting pods on unready nodes to 1 minute
+  * Respond more quickly to node preemption (previously 5 minutes)
+* Fix automatic worker deletion on shutdown for cloud platforms
+  * Lowering Kubelet privileges in [#372](https://github.com/poseidon/typhoon/pull/372) dropped a needed node deletion authorization. Scale-in due to manual terraform apply (any cloud), AWS spot termination, or Azure low priority deletion left old nodes registered, requiring manual deletion (`kubectl delete node name`)
+
+#### AWS
+
+* Add `ingress_zone_id` output with the NLB DNS name's Route53 zone for use in alias records ([#380](https://github.com/poseidon/typhoon/pull/380))
+
+#### Azure
+
+* Fix azure provider warning, `public_ip` `allocation_method` replaces `public_ip_address_allocation`
+  * Require `terraform-provider-azurerm` v1.21+ (action required)
+
+#### Addons
+
+* Update nginx-ingress from v0.21.0 to v0.22.0
+* Update Prometheus from v2.6.0 to v2.7.1
+* Update kube-state-metrics from v1.4.0 to v1.5.0
+  * Fix ClusterRole to collect and export PodDisruptionBudget metrics ([#383](https://github.com/poseidon/typhoon/pull/383))
+* Update node-exporter from v0.15.2 to v0.17.0
+* Update Grafana from v5.4.2 to v5.4.3
+
+## v1.13.2
+
+* Kubernetes [v1.13.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1132)
+* Add ServiceAccounts for `kube-apiserver` and `kube-scheduler` ([#370](https://github.com/poseidon/typhoon/pull/370))
+* Use lower-privilege TLS client certificates for Kubelets ([#372](https://github.com/poseidon/typhoon/pull/372))
+* Use HTTPS liveness probes for `kube-scheduler` and `kube-controller-manager` ([#377](https://github.com/poseidon/typhoon/pull/377))
+* Update CoreDNS from v1.2.6 to [v1.3.0](https://coredns.io/2018/12/15/coredns-1.3.0-release/)
+* Allow the `certificates.k8s.io` API to issue certificates signed by the cluster CA ([#376](https://github.com/poseidon/typhoon/pull/376))
+  * Configure controller manager to sign CSRs that are manually [approved](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster) by an administrator
+
+#### AWS
+
+* Change `controller_type` and `worker_type` default from t2.small to t3.small ([#365](https://github.com/poseidon/typhoon/pull/365))
+  * t3.small is cheaper, provides 2 vCPU (instead of 1), and 5 Gbps of pod-to-pod bandwidth!
+
+#### Bare-Metal
+
+* Remove the `kubeconfig` output variable
+
+#### Addons
+
+* Update Prometheus from v2.5.0 to v2.6.0
+
+## v1.13.1
+
+* Kubernetes [v1.13.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1131)
+* Update Calico from v3.3.2 to [v3.4.0](https://docs.projectcalico.org/v3.4/releases/) ([#362](https://github.com/poseidon/typhoon/pull/362))
+  * Install CNI plugins with an init container rather than a sidecar
+  * Improve the `calico-node` ClusterRole
+* Recommend updating `terraform-provider-ct` plugin from v0.2.1 to v0.3.0 ([#363](https://github.com/poseidon/typhoon/pull/363))
+  * [Migration](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct) instructions for upgrading `terraform-provider-ct` in-place for v1.12.2+ clusters (**action required**)
+  * [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`
+  * Require Container Linux 1688.5.3 or newer
+
+#### Google Cloud
+
+* Increase TCP proxy apiserver backend service timeout from 1 minute to 5 minutes ([#361](https://github.com/poseidon/typhoon/pull/361))
+  * Align `port-forward` behavior closer to AWS/Azure (no timeout)
+
+#### Addons
+
+* Update Grafana from v5.4.0 to v5.4.2
+
+## v1.13.0
+
+* Kubernetes [v1.13.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1130)
+* Update Calico from v3.3.1 to [v3.3.2](https://docs.projectcalico.org/v3.3/releases/)
+
+#### Addons
+
+* Update Grafana from v5.3.4 to v5.4.0
+* Disable Grafana login form, since admin user can't be disabled ([#352](https://github.com/poseidon/typhoon/pull/352))
+  * Example manifests aim to provide a read-only dashboard view
+
+## v1.12.3
+
+* Kubernetes [v1.12.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1123)
+* Add `enable_reporting` variable (default "false") to provide upstreams with usage data ([#345](https://github.com/poseidon/typhoon/pull/345))
+* Change kube-apiserver `--kubelet-preferred-address-types` to InternalIP,ExternalIP,Hostname
+* Update Calico from v3.3.0 to [v3.3.1](https://docs.projectcalico.org/v3.3/releases/)
+  * Disable Felix usage reporting by default ([#345](https://github.com/poseidon/typhoon/pull/345))
+* Improve flannel manifests
+  * [Rename](https://github.com/poseidon/terraform-render-bootkube/commit/d045a8e6b8eccfbb9d69bb51953b5a93d23f67f7) `kube-flannel` DaemonSet to `flannel` and `kube-flannel-cfg` ConfigMap to `flannel-config` 
+  * [Drop](https://github.com/poseidon/terraform-render-bootkube/commit/39f9afb3360ec642e5b98457c8bd07eda35b6c96) unused mounts and add a CPU resource request
+* Update CoreDNS from v1.2.4 to [v1.2.6](https://coredns.io/2018/11/05/coredns-1.2.6-release/)
+  * Enable CoreDNS `loop` and `loadbalance` plugins ([#340](https://github.com/poseidon/typhoon/pull/340))
+* Fix pod-checkpointer log noise and checkpointable pods detection ([#346](https://github.com/poseidon/typhoon/pull/346))
+* Use kubernetes-incubator/bootkube v0.14.0
+* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`.
+  * Allows pinning `terraform-provider-ct` and `terraform-provider-matchbox` versions
+  * Improves safety of later plugin version migrations
+
+#### Azure
+
+* Use eviction policy `Delete` for `Low` priority virtual machine scale set workers ([#343](https://github.com/poseidon/typhoon/pull/343))
+  * Fix issue where Azure defaults to `Deallocate` eviction policy, which required manually restarting deallocated instances. `Delete` policy aligns Azure with AWS and GCP behavior.
+  * Require `terraform-provider-azurerm` v1.19+ (action required)
+
+#### Bare-Metal
+
+* Add Kubelet `/etc/iscsi` and `iscsadm` mounts on bare-metal for iSCSI ([#103](https://github.com/poseidon/typhoon/pull/103))
+
+#### Addons
+
+* Update nginx-ingress from v0.20.0 to v0.21.0
+* Update Prometheus from v2.4.3 to v2.5.0
+* Update Grafana from v5.3.2 to v5.3.4
+
 ## v1.12.2
 
 * Kubernetes [v1.12.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122)

README.md

@@ -11,29 +11,32 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Modules
 
-Typhoon provides a Terraform Module for each supported operating system and platform.
+Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.
 
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
-| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
 
-The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
+Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
+| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
+| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
 
 ## Documentation
 
@@ -47,7 +50,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
   
   providers = {
     google   = "google.default"
@@ -87,10 +90,10 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 ```sh
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
 ```
 
 List the pods.
@@ -102,6 +105,7 @@ kube-system   calico-node-1cs8z                         2/2    Running   0
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
@@ -111,6 +115,7 @@ kube-system   kube-proxy-njn47                          1/1    Running   0
 kube-system   kube-scheduler-3895335239-5x87r           1/1    Running   0         6m
 kube-system   kube-scheduler-3895335239-bzrrt           1/1    Running   1         6m
 kube-system   pod-checkpointer-l6lrt                    1/1    Running   0         6m
+kube-system   pod-checkpointer-l6lrt-controller-0       1/1    Running   0         6m
 ```
 
 ## Non-Goals

addons/grafana/dashboards.yaml

@@ -1963,7 +1963,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_cpu{mode=\"idle\"}[2m])) * 100",
+                  "expr": "sum(rate(node_cpu_seconds_total{mode=\"idle\"}[2m])) * 100",
                   "hide": false,
                   "intervalFactor": 10,
                   "legendFormat": "",
@@ -2138,7 +2138,7 @@ data:
               "renderer": "flot",
               "seriesOverrides": [
                 {
-                  "alias": "node_memory_SwapFree{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
+                  "alias": "node_memory_SwapFree_bytes{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
                   "yaxis": 2
                 }
               ],
@@ -2148,7 +2148,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_memory_MemTotal) - sum(node_memory_MemFree) - sum(node_memory_Buffers) - sum(node_memory_Cached)",
+                  "expr": "sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes) - sum(node_memory_Buffers_bytes) - sum(node_memory_Cached_bytes)",
                   "intervalFactor": 2,
                   "legendFormat": "memory usage",
                   "metric": "memo",
@@ -2157,7 +2157,7 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(node_memory_Buffers)",
+                  "expr": "sum(node_memory_Buffers_bytes)",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory buffers",
@@ -2167,7 +2167,7 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(node_memory_Cached)",
+                  "expr": "sum(node_memory_Cached_bytes)",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory cached",
@@ -2177,7 +2177,7 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(node_memory_MemFree)",
+                  "expr": "sum(node_memory_MemFree_bytes)",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory free",
@@ -2268,7 +2268,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "((sum(node_memory_MemTotal) - sum(node_memory_MemFree) - sum(node_memory_Buffers) - sum(node_memory_Cached)) / sum(node_memory_MemTotal)) * 100",
+                  "expr": "((sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes) - sum(node_memory_Buffers_bytes) - sum(node_memory_Cached_bytes)) / sum(node_memory_MemTotal_bytes)) * 100",
                   "intervalFactor": 2,
                   "metric": "",
                   "refId": "A",
@@ -2355,7 +2355,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_disk_bytes_read[5m]))",
+                  "expr": "max(rate(node_disk_read_bytes_total[5m]))",
                   "hide": false,
                   "intervalFactor": 4,
                   "legendFormat": "read",
@@ -2364,14 +2364,14 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(rate(node_disk_bytes_written[5m]))",
+                  "expr": "max(rate(node_disk_written_bytes_total[5m]))",
                   "intervalFactor": 4,
                   "legendFormat": "written",
                   "refId": "B",
                   "step": 20
                 },
                 {
-                  "expr": "sum(rate(node_disk_io_time_ms[5m]))",
+                  "expr": "max(rate(node_disk_io_time_seconds_total[5m]))",
                   "intervalFactor": 4,
                   "legendFormat": "io time",
                   "refId": "C",
@@ -2458,7 +2458,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "(sum(node_filesystem_size{device!=\"rootfs\"}) - sum(node_filesystem_free{device!=\"rootfs\"})) / sum(node_filesystem_size{device!=\"rootfs\"})",
+                  "expr": "(sum(node_filesystem_size_bytes{device!=\"rootfs\"}) - sum(node_filesystem_free_bytes{device!=\"rootfs\"})) / sum(node_filesystem_size_bytes{device!=\"rootfs\"})",
                   "intervalFactor": 2,
                   "refId": "A",
                   "step": 60,
@@ -2536,7 +2536,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_network_receive_bytes{device!~\"lo\"}[5m]))",
+                  "expr": "sum(rate(node_network_receive_bytes_total{device!~\"lo\"}[5m]))",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -2618,7 +2618,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_network_transmit_bytes{device!~\"lo\"}[5m]))",
+                  "expr": "sum(rate(node_network_transmit_bytes_total{device!~\"lo\"}[5m]))",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -4093,7 +4093,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "sum(100 - (avg by (instance) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[5m])) * 100)) / count(node_cpu{job=\"node-exporter\",mode=\"idle\"})",
+                  "expr": "sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[5m])) * 100)) / count(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "refId": "A",
@@ -4165,7 +4165,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "((sum(node_memory_MemTotal) - sum(node_memory_MemFree) - sum(node_memory_Buffers) - sum(node_memory_Cached)) / sum(node_memory_MemTotal)) * 100",
+                  "expr": "((sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes) - sum(node_memory_Buffers_bytes) - sum(node_memory_Cached_bytes)) / sum(node_memory_MemTotal_bytes)) * 100",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "refId": "A",
@@ -4237,7 +4237,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "(sum(node_filesystem_size{device!=\"rootfs\"}) - sum(node_filesystem_free{device!=\"rootfs\"})) / sum(node_filesystem_size{device!=\"rootfs\"})",
+                  "expr": "(sum(node_filesystem_size_bytes{device!=\"rootfs\"}) - sum(node_filesystem_free_bytes{device!=\"rootfs\"})) / sum(node_filesystem_size_bytes{device!=\"rootfs\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "refId": "A",
@@ -5476,7 +5476,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "100 - (avg by (cpu) (irate(node_cpu{mode=\"idle\", instance=\"$server\"}[5m])) * 100)",
+                  "expr": "100 - (avg by (cpu) (irate(node_cpu_seconds_total{mode=\"idle\", instance=\"$server\"}[5m])) * 100)",
                   "hide": false,
                   "intervalFactor": 10,
                   "legendFormat": "{{cpu}}",
@@ -5652,7 +5652,7 @@ data:
               "renderer": "flot",
               "seriesOverrides": [
                 {
-                  "alias": "node_memory_SwapFree{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
+                  "alias": "node_memory_SwapFree_bytes{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
                   "yaxis": 2
                 }
               ],
@@ -5662,7 +5662,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "node_memory_MemTotal{instance=\"$server\"} - node_memory_MemFree{instance=\"$server\"} - node_memory_Buffers{instance=\"$server\"} - node_memory_Cached{instance=\"$server\"}",
+                  "expr": "node_memory_MemTotal_bytes{instance=\"$server\"} - node_memory_MemFree_bytes{instance=\"$server\"} - node_memory_Buffers_bytes{instance=\"$server\"} - node_memory_Cached_bytes{instance=\"$server\"}",
                   "hide": false,
                   "interval": "",
                   "intervalFactor": 2,
@@ -5672,7 +5672,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "node_memory_Buffers{instance=\"$server\"}",
+                  "expr": "node_memory_Buffers_bytes{instance=\"$server\"}",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory buffers",
@@ -5689,7 +5689,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "node_memory_MemFree{instance=\"$server\"}",
+                  "expr": "node_memory_MemFree_bytes{instance=\"$server\"}",
                   "intervalFactor": 2,
                   "legendFormat": "memory free",
                   "metric": "",
@@ -5778,7 +5778,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "((node_memory_MemTotal{instance=\"$server\"} - node_memory_MemFree{instance=\"$server\"}  - node_memory_Buffers{instance=\"$server\"} - node_memory_Cached{instance=\"$server\"}) / node_memory_MemTotal{instance=\"$server\"}) * 100",
+                  "expr": "((node_memory_MemTotal_bytes{instance=\"$server\"} - node_memory_MemFree_bytes{instance=\"$server\"}  - node_memory_Buffers_bytes{instance=\"$server\"} - node_memory_Cached_bytes{instance=\"$server\"}) / node_memory_MemTotal_bytes{instance=\"$server\"}) * 100",
                   "intervalFactor": 2,
                   "refId": "A",
                   "step": 60,
@@ -5864,7 +5864,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum by (instance) (rate(node_disk_bytes_read{instance=\"$server\"}[2m]))",
+                  "expr": "sum by (instance) (rate(node_disk_read_bytes_total{instance=\"$server\"}[2m]))",
                   "hide": false,
                   "intervalFactor": 4,
                   "legendFormat": "read",
@@ -5873,14 +5873,14 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum by (instance) (rate(node_disk_bytes_written{instance=\"$server\"}[2m]))",
+                  "expr": "sum by (instance) (rate(node_disk_written_bytes_total{instance=\"$server\"}[2m]))",
                   "intervalFactor": 4,
                   "legendFormat": "written",
                   "refId": "B",
                   "step": 20
                 },
                 {
-                  "expr": "sum by (instance) (rate(node_disk_io_time_ms{instance=\"$server\"}[2m]))",
+                  "expr": "sum by (instance) (rate(node_disk_io_time_seconds_total{instance=\"$server\"}[2m]))",
                   "intervalFactor": 4,
                   "legendFormat": "io time",
                   "refId": "C",
@@ -5967,7 +5967,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "(sum(node_filesystem_size{device!=\"rootfs\",instance=\"$server\"}) - sum(node_filesystem_free{device!=\"rootfs\",instance=\"$server\"})) / sum(node_filesystem_size{device!=\"rootfs\",instance=\"$server\"})",
+                  "expr": "(sum(node_filesystem_size_bytes{device!=\"rootfs\",instance=\"$server\"}) - sum(node_filesystem_free_bytes{device!=\"rootfs\",instance=\"$server\"})) / sum(node_filesystem_size_bytes{device!=\"rootfs\",instance=\"$server\"})",
                   "intervalFactor": 2,
                   "refId": "A",
                   "step": 60,
@@ -6045,7 +6045,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_network_receive_bytes{instance=\"$server\",device!~\"lo\"}[5m])",
+                  "expr": "rate(node_network_receive_bytes_total{instance=\"$server\",device!~\"lo\"}[5m])",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "{{device}}",
@@ -6127,7 +6127,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_network_transmit_bytes{instance=\"$server\",device!~\"lo\"}[5m])",
+                  "expr": "rate(node_network_transmit_bytes_total{instance=\"$server\",device!~\"lo\"}[5m])",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "{{device}}",
@@ -6184,7 +6184,7 @@ data:
             "multi": false,
             "name": "server",
             "options": [],
-            "query": "label_values(node_boot_time, instance)",
+            "query": "label_values(node_boot_time_seconds, instance)",
             "refresh": 1,
             "regex": "",
             "sort": 0,

addons/grafana/deployment.yaml

@@ -23,12 +23,14 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:5.3.2
+          image: grafana/grafana:5.4.3
           env:
             - name: GF_SERVER_HTTP_PORT
               value: "8080"
             - name: GF_AUTH_BASIC_ENABLED
               value: "false"
+            - name: GF_AUTH_DISABLE_LOGIN_FORM
+              value: "true"
             - name: GF_AUTH_ANONYMOUS_ENABLED
               value: "true"
             - name: GF_AUTH_ANONYMOUS_ORG_ROLE

addons/nginx-ingress/aws/deployment.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/azure/deployment.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -22,7 +22,7 @@ spec:
     spec:
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/prometheus/config.yaml

@@ -102,7 +102,7 @@ data:
           regex: 'true'
         - action: labelmap
           regex: __meta_kubernetes_node_label_(.+)
-        - source_labels: [__meta_kubernetes_node_name]
+        - source_labels: [__meta_kubernetes_node_address_InternalIP]
           action: replace
           target_label: __address__
           replacement: '${1}:2381'

addons/prometheus/deployment.yaml

@@ -20,7 +20,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.4.3
+          image: quay.io/prometheus/prometheus:v2.7.1
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml

@@ -3,7 +3,8 @@ kind: ClusterRole
 metadata:
   name: kube-state-metrics
 rules:
-- apiGroups: [""]
+- apiGroups:
+  - ""
   resources:
   - configmaps
   - secrets
@@ -17,23 +18,47 @@ rules:
   - persistentvolumes
   - namespaces
   - endpoints
-  verbs: ["list", "watch"]
-- apiGroups: ["extensions"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - extensions
   resources:
   - daemonsets
   - deployments
   - replicasets
-  verbs: ["list", "watch"]
-- apiGroups: ["apps"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - apps
   resources:
   - statefulsets
-  verbs: ["list", "watch"]
-- apiGroups: ["batch"]
+  - daemonsets
+  - deployments
+  - replicasets
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - batch
   resources:
   - cronjobs
   - jobs
-  verbs: ["list", "watch"]
-- apiGroups: ["autoscaling"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - autoscaling
   resources:
   - horizontalpodautoscalers
-  verbs: ["list", "watch"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - policy
+  resources:
+  - poddisruptionbudgets
+  verbs:
+  - list
+  - watch

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -24,7 +24,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.4.0
+        image: quay.io/coreos/kube-state-metrics:v1.5.0
         ports:
           - name: metrics
             containerPort: 8080
@@ -35,7 +35,7 @@ spec:
           initialDelaySeconds: 5
           timeoutSeconds: 5
       - name: addon-resizer
-        image: k8s.gcr.io/addon-resizer:1.7
+        image: k8s.gcr.io/addon-resizer:1.8.4
         resources:
           limits:
             cpu: 100m

addons/prometheus/exporters/kube-state-metrics/resizer-role-binding.yaml

@@ -6,7 +6,7 @@ metadata:
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role
-  name: kube-state-metrics-resizer
+  name: kube-state-metrics
 subjects:
 - kind: ServiceAccount
   name: kube-state-metrics

addons/prometheus/exporters/kube-state-metrics/resizer-role.yaml

@@ -1,15 +1,31 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
-  name: kube-state-metrics-resizer
+  name: kube-state-metrics
   namespace: monitoring
 rules:
-- apiGroups: [""]
+- apiGroups:
+  - ""
   resources:
   - pods
-  verbs: ["get"]
-- apiGroups: ["extensions"]
+  verbs:
+  - get
+- apiGroups:
+  - extensions
   resources:
   - deployments
-  resourceNames: ["kube-state-metrics"]
-  verbs: ["get", "update"]
+  resourceNames:
+  - kube-state-metrics
+  verbs:
+  - get
+  - update
+- apiGroups:
+  - apps
+  resources:
+  - deployments
+  resourceNames:
+  - kube-state-metrics
+  verbs:
+  - get
+  - update
+

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -28,21 +28,24 @@ spec:
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v0.15.2
+        image: quay.io/prometheus/node-exporter:v0.17.0
         args:
-          - "--path.procfs=/host/proc"
-          - "--path.sysfs=/host/sys"
+          - --path.procfs=/host/proc
+          - --path.sysfs=/host/sys
+          - --path.rootfs=/host/root
+          - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
+          - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
         ports:
           - name: metrics
             containerPort: 9100
             hostPort: 9100
         resources:
           requests:
-            memory: 30Mi
             cpu: 100m
-          limits:
             memory: 50Mi
+          limits:
             cpu: 200m
+            memory: 100Mi
         volumeMounts:
           - name: proc
             mountPath: /host/proc
@@ -50,6 +53,9 @@ spec:
           - name: sys
             mountPath: /host/sys
             readOnly: true
+          - name: root
+            mountPath: /host/root
+            readOnly: true
       tolerations:
         - effect: NoSchedule
           operator: Exists
@@ -60,3 +66,6 @@ spec:
         - name: sys
           hostPath:
             path: /sys
+        - name: root
+          hostPath:
+            path: /

addons/prometheus/rules.yaml

@@ -456,22 +456,22 @@ data:
     - name: node.rules
       rules:
       - record: instance:node_cpu:rate:sum
-        expr: sum(rate(node_cpu{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
+        expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
           BY (instance)
       - record: instance:node_filesystem_usage:sum
-        expr: sum((node_filesystem_size{mountpoint="/"} - node_filesystem_free{mountpoint="/"}))
+        expr: sum((node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"}))
           BY (instance)
       - record: instance:node_network_receive_bytes:rate:sum
-        expr: sum(rate(node_network_receive_bytes[3m])) BY (instance)
+        expr: sum(rate(node_network_receive_bytes_total[3m])) BY (instance)
       - record: instance:node_network_transmit_bytes:rate:sum
-        expr: sum(rate(node_network_transmit_bytes[3m])) BY (instance)
+        expr: sum(rate(node_network_transmit_bytes_total[3m])) BY (instance)
       - record: instance:node_cpu:ratio
-        expr: sum(rate(node_cpu{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
-          GROUP_LEFT() count(sum(node_cpu) BY (instance, cpu)) BY (instance)
+        expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
+          GROUP_LEFT() count(sum(node_cpu_seconds_total) BY (instance, cpu)) BY (instance)
       - record: cluster:node_cpu:sum_rate5m
-        expr: sum(rate(node_cpu{mode!="idle"}[5m]))
+        expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
       - record: cluster:node_cpu:ratio
-        expr: cluster:node_cpu:rate5m / count(sum(node_cpu) BY (instance, cpu))
+        expr: cluster:node_cpu:sum_rate5m / count(sum(node_cpu_seconds_total) BY (instance, cpu))
       - alert: NodeExporterDown
         expr: absent(up{job="node-exporter"} == 1)
         for: 10m
@@ -481,7 +481,7 @@ data:
           description: Prometheus could not scrape a node-exporter for more than 10m,
             or node-exporters have disappeared from discovery
       - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free[6h], 3600 * 24) < 0
+        expr: predict_linear(node_filesystem_free_bytes[6h], 3600 * 24) < 0
         for: 30m
         labels:
           severity: warning
@@ -489,7 +489,7 @@ data:
           description: device {{$labels.device}} on node {{$labels.instance}} is running
             full within the next 24 hours (mounted at {{$labels.mountpoint}})
       - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free[30m], 3600 * 2) < 0
+        expr: predict_linear(node_filesystem_free_bytes[30m], 3600 * 2) < 0
         for: 10m
         labels:
           severity: critical

aws/container-linux/kubernetes/README.md

@@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,4 +11,5 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -78,7 +78,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -143,17 +143,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

aws/container-linux/kubernetes/controllers.tf

@@ -68,10 +68,10 @@ data "template_file" "controller-configs" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

aws/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
@@ -5,6 +9,11 @@ output "ingress_dns_name" {
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_zone_id" {
+  value       = "${aws_lb.nlb.zone_id}"
+  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
+}
+
 # Outputs for worker pools
 
 output "vpc_id" {
@@ -23,7 +32,7 @@ output "worker_security_groups" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }
 
 # Outputs for custom load balancing

aws/container-linux/kubernetes/variables.tf

@@ -31,13 +31,13 @@ variable "worker_count" {
 
 variable "controller_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 
@@ -134,3 +134,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

aws/container-linux/kubernetes/workers.tf

@@ -13,7 +13,7 @@ module "workers" {
   spot_price      = "${var.worker_price}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -51,7 +51,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.2 \
+            docker://k8s.gcr.io/hyperkube:v1.13.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/variables.tf

@@ -30,7 +30,7 @@ variable "count" {
 
 variable "instance_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type"
 }
 

aws/container-linux/kubernetes/workers/workers.tf

@@ -77,9 +77,9 @@ data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

aws/fedora-atomic/kubernetes/README.md

@@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,6 +11,7 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 
   # Fedora
   trusted_certs_dir = "/etc/pki/tls/certs"

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -19,24 +19,9 @@ write_files:
       ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
       ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
       ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/latest/meta-data/local-ipv4\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -55,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -93,11 +78,10 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, cloud-metadata.service]
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/controllers.tf

@@ -60,10 +60,10 @@ data "template_file" "controller-cloudinit" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(6, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(6, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

aws/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
@@ -5,6 +9,11 @@ output "ingress_dns_name" {
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_zone_id" {
+  value       = "${aws_lb.nlb.zone_id}"
+  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
+}
+
 # Outputs for worker pools
 
 output "vpc_id" {
@@ -23,7 +32,7 @@ output "worker_security_groups" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }
 
 # Outputs for custom load balancing

aws/fedora-atomic/kubernetes/variables.tf

@@ -31,13 +31,13 @@ variable "worker_count" {
 
 variable "controller_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 
@@ -116,3 +116,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

aws/fedora-atomic/kubernetes/workers.tf

@@ -12,7 +12,7 @@ module "workers" {
   spot_price      = "${var.worker_price}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -1,23 +1,8 @@
 #cloud-config
 write_files:
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/latest/meta-data/local-ipv4\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -34,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -69,8 +54,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -30,7 +30,7 @@ variable "count" {
 
 variable "instance_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type"
 }
 

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -70,9 +70,9 @@ data "template_file" "worker-cloudinit" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(6, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(6, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

azure/container-linux/kubernetes/README.md

@@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

azure/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -10,4 +10,5 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -78,7 +78,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -123,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -143,17 +143,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

azure/container-linux/kubernetes/controllers.tf

@@ -121,10 +121,10 @@ resource "azurerm_public_ip" "controllers" {
   count               = "${var.controller_count}"
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-controller-${count.index}"
-  location                     = "${azurerm_resource_group.cluster.location}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-controller-${count.index}"
+  location          = "${azurerm_resource_group.cluster.location}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Controller Ignition configs
@@ -149,10 +149,10 @@ data "template_file" "controller-configs" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

azure/container-linux/kubernetes/lb.tf

@@ -17,20 +17,20 @@ resource "azurerm_dns_a_record" "apiserver" {
 resource "azurerm_public_ip" "apiserver-ipv4" {
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-apiserver-ipv4"
-  location                     = "${var.region}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-apiserver-ipv4"
+  location          = "${var.region}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Static IPv4 address for the ingress frontend
 resource "azurerm_public_ip" "ingress-ipv4" {
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-ingress-ipv4"
-  location                     = "${var.region}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-ingress-ipv4"
+  location          = "${var.region}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress

azure/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
@@ -28,5 +32,5 @@ output "backend_address_pool_id" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }

azure/container-linux/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "azurerm" {
-  version = "~> 1.17"
+  version = "~> 1.21"
 }
 
 provider "local" {

azure/container-linux/kubernetes/variables.tf

@@ -115,3 +115,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

azure/container-linux/kubernetes/workers.tf

@@ -15,7 +15,7 @@ module "workers" {
   priority = "${var.worker_priority}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -51,7 +51,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -93,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -111,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.2 \
+            docker://k8s.gcr.io/hyperkube:v1.13.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')

azure/container-linux/kubernetes/workers/workers.tf

@@ -67,8 +67,9 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
   }
 
   # lifecycle
-  priority            = "${var.priority}"
   upgrade_policy_mode = "Manual"
+  priority            = "${var.priority}"
+  eviction_policy     = "Delete"
 }
 
 # Scale up or down to maintain desired number, tolerating deallocations.
@@ -105,9 +106,9 @@ data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

bare-metal/container-linux/kubernetes/README.md

@@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name                    = "${var.cluster_name}"
   api_servers                     = ["${var.k8s_domain_name}"]
@@ -12,4 +12,5 @@ module "bootkube" {
   pod_cidr                        = "${var.pod_cidr}"
   service_cidr                    = "${var.service_cidr}"
   cluster_domain_suffix           = "${var.cluster_domain_suffix}"
+  enable_reporting                = "${var.enable_reporting}"
 }

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@@ -70,6 +70,10 @@ systemd:
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -86,7 +90,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -124,7 +128,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/hostname
       filesystem: root
       mode: 0644
@@ -150,17 +154,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$BOOTKUBE_ASSETS \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $$RKT_OPTS \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -45,6 +45,10 @@ systemd:
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -59,7 +63,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -85,7 +89,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,3 @@
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
 }

bare-metal/container-linux/kubernetes/profiles.tf

@@ -160,12 +160,12 @@ data "template_file" "controller-configs" {
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.controller_domains, count.index)}"
-    etcd_name             = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.controller_domains, count.index)}"
+    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 
@@ -191,10 +191,10 @@ data "template_file" "worker-configs" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.worker_domains, count.index)}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 

bare-metal/container-linux/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -94,7 +94,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

bare-metal/container-linux/kubernetes/variables.tf

@@ -141,3 +141,9 @@ variable "kernel_args" {
   type        = "list"
   default     = []
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

bare-metal/fedora-atomic/kubernetes/README.md

@@ -11,8 +11,8 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${var.k8s_domain_name}"]
@@ -11,6 +11,7 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 
   # Fedora
   trusted_certs_dir = "/etc/pki/tls/certs"

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -40,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -84,9 +84,9 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -19,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -60,7 +60,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

bare-metal/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,3 @@
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
 }

bare-metal/fedora-atomic/kubernetes/profiles.tf

@@ -55,12 +55,12 @@ data "template_file" "controller-configs" {
   template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.controller_domains, count.index)}"
-    etcd_name             = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.controller_domains, count.index)}"
+    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 
@@ -79,9 +79,9 @@ data "template_file" "worker-configs" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.worker_domains, count.index)}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }

bare-metal/fedora-atomic/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -92,7 +92,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

bare-metal/fedora-atomic/kubernetes/variables.tf

@@ -110,3 +110,9 @@ variable "kernel_args" {
   type        = "list"
   default     = []
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

digital-ocean/container-linux/kubernetes/README.md

@@ -11,10 +11,11 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, and other [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,4 +11,5 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -56,12 +56,9 @@ systemd:
       contents: |
         [Unit]
         Description=Kubelet via Hyperkube
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
         EnvironmentFile=/etc/kubernetes/kubelet.env
-        EnvironmentFile=/run/metadata/coreos
         Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
           --volume=resolv,kind=host,source=/etc/resolv.conf \
           --mount volume=resolv,target=/etc/resolv.conf \
@@ -89,11 +86,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/etc/kubernetes/kubeconfig \
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
@@ -129,7 +125,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -149,17 +145,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -31,12 +31,9 @@ systemd:
       contents: |
         [Unit]
         Description=Kubelet via Hyperkube
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
         EnvironmentFile=/etc/kubernetes/kubelet.env
-        EnvironmentFile=/run/metadata/coreos
         Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
           --volume=resolv,kind=host,source=/etc/resolv.conf \
           --mount volume=resolv,target=/etc/resolv.conf \
@@ -62,11 +59,10 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/etc/kubernetes/kubeconfig \
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
@@ -99,7 +95,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.2
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -117,7 +113,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.2 \
+            docker://k8s.gcr.io/hyperkube:v1.13.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/container-linux/kubernetes/controllers.tf

@@ -83,9 +83,9 @@ data "template_file" "controller-configs" {
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
 
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster  = "${join(",", data.template_file.etcds.*.rendered)}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    etcd_initial_cluster   = "${join(",", data.template_file.etcds.*.rendered)}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

digital-ocean/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 output "controllers_dns" {
   value = "${digitalocean_record.controllers.0.fqdn}"
 }

digital-ocean/container-linux/kubernetes/ssh.tf

@@ -10,7 +10,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -78,7 +78,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/container-linux/kubernetes/variables.tf

@@ -92,3 +92,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

digital-ocean/container-linux/kubernetes/workers.tf

@@ -66,7 +66,7 @@ data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

digital-ocean/fedora-atomic/kubernetes/README.md

@@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

digital-ocean/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=f39f8294c465397e622c606174e6f412ee3ca0f8"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,6 +11,7 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 
   # Fedora
   trusted_certs_dir = "/etc/pki/tls/certs"

digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -19,24 +19,9 @@ write_files:
       ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
       ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
       ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -55,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -90,11 +75,10 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, cloud-metadata.service]
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -1,23 +1,8 @@
 #cloud-config
 write_files:
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -34,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -66,8 +51,7 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/controllers.tf

@@ -77,9 +77,9 @@ data "template_file" "controller-cloudinit" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

digital-ocean/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 output "controllers_dns" {
   value = "${digitalocean_record.controllers.0.fqdn}"
 }

digital-ocean/fedora-atomic/kubernetes/ssh.tf

@@ -10,7 +10,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -76,7 +76,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/fedora-atomic/kubernetes/variables.tf

@@ -85,3 +85,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

digital-ocean/fedora-atomic/kubernetes/workers.tf

@@ -59,8 +59,8 @@ data "template_file" "worker-cloudinit" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars = {
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

docs/addons/heapster.md

@@ -1,6 +1,6 @@
 # Heapster
 
-[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command and Kubernetes dashboard graphs.
+[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command.
 
 ## Create
 

docs/advanced/worker-pools.md

@@ -16,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo
 
 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.13.3"
   
   providers = {
     aws = "aws.default"
@@ -67,7 +67,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
 | count | Number of instances | 1 | 3 |
-| instance_type | EC2 instance type | "t2.small" | "t2.medium" |
+| instance_type | EC2 instance type | "t3.small" | "t3.medium" |
 | os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | spot_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
@@ -82,7 +82,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define
 
 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.13.3"
   
   providers = {
     azurerm = "azurerm.default"
@@ -152,7 +152,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu
 
 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.13.3"
 
   providers = {
     google = "google.default"
@@ -187,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.12.2
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.12.2
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.12.2
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.12.2
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.12.2
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.13.3
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.13.3
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.13.3
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.13.3
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.13.3
 ```
 
 ### Variables

docs/announce.md

@@ -18,7 +18,7 @@ Fedora Atomic is a container-optimized operating system designed for large-scale
 
 For newcomers, Typhoon is a free (cost and freedom) Kubernetes distribution providing upstream Kubernetes, declarative configuration via [Terraform](https://www.terraform.io/intro/index.html), and support for AWS, Google Cloud, DigitalOcean, and bare-metal. Typhoon clusters use a [self-hosted](https://github.com/kubernetes-incubator/bootkube) control plane, support [Calico](https://www.projectcalico.org/blog/) and [flannel](https://coreos.com/flannel/docs/latest/) CNI networking, and enable etcd TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/), and network policy.
 
-Typhoon for Fedora Atomic reflects many of the same principles that created Typhoon for Container Linux. Clusters are declared using plain Terraform configs that can be versioned. In lieu of Ignition, instances are declaratively provisioned with Cloud-Init and kickstart (bare-metal only). TLS assets are generated. Hosts run only a kubelet service, other components are scheduled (i.e. self-hosted). The upstream hyperkube is used directly[^1]. And clusters are kept minimal by offering optional addons for [Ingress](https://typhoon.psdn.io/addons/ingress/), [Prometheus](https://typhoon.psdn.io/addons/prometheus/), and [Grafana](https://typhoon.psdn.io/addons/grafana/). Typhoon compliments and enhances Fedora Atomic as a choice of operating system for Kubernetes.
+Typhoon for Fedora Atomic reflects many of the same principles that created Typhoon for Container Linux. Clusters are declared using plain Terraform configs that can be versioned. In lieu of Ignition, instances are declaratively provisioned with Cloud-Init and kickstart (bare-metal only). TLS assets are generated. Hosts run only a kubelet service, other components are scheduled (i.e. self-hosted). The upstream hyperkube is used directly[^1]. And clusters are kept minimal by offering optional addons for [Ingress](/addons/ingress/), [Prometheus](/addons/prometheus/), and [Grafana](/addons/grafana/). Typhoon compliments and enhances Fedora Atomic as a choice of operating system for Kubernetes.
 
 Meanwhile, Fedora Atomic adds some promising new low-level technologies:
 

docs/architecture/concepts.md

@@ -69,7 +69,7 @@ Module versioning ensures `terraform get --update` only fetches the desired vers
 
 Maintain Terraform configs for "live" infrastructure in a versioned repository. Seek to organize configs to reflect resources that should be managed together in a `terraform apply` invocation.
 
-You may choose to organize resources all together, by team, by project, or some other scheme. Here's an example that manages four clusters together:
+You may choose to organize resources all together, by team, by project, or some other scheme. Here's an example that manages clusters together:
 
 ```sh
 .git/

docs/architecture/digitalocean.md

@@ -1,4 +1,4 @@
-# AWS
+# DigitalOcean
 
 ## IPv6
 

docs/atomic/aws.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on AWS with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on AWS with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `aws/fedora-atomic/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.13.3"
 
   providers = {
     aws = "aws.default"
@@ -155,10 +155,10 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.12.2
-ip-10-0-19-112   Ready     34m       v1.12.2
-ip-10-0-4-22     Ready     34m       v1.12.2
+NAME           STATUS  ROLES              AGE  VERSION
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.3
+ip-10-0-26-65  Ready   node               10m  v1.13.3
+ip-10-0-41-21  Ready   node               10m  v1.13.3
 ```
 
 List the pods.
@@ -170,6 +170,7 @@ kube-system   calico-node-1m5bf                         2/2    Running   0
 kube-system   calico-node-7jmr1                         2/2    Running   0         34m              
 kube-system   calico-node-bknc8                         2/2    Running   0         34m              
 kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m              
+kube-system   coredns-1187388186-qjnvp                  1/1    Running   0         34m
 kube-system   kube-apiserver-4mjbk                      1/1    Running   0         34m              
 kube-system   kube-controller-manager-3597210155-j2jbt  1/1    Running   1         34m              
 kube-system   kube-controller-manager-3597210155-j7g7x  1/1    Running   0         34m              
@@ -179,7 +180,7 @@ kube-system   kube-proxy-sbbsh                          1/1    Running   0
 kube-system   kube-scheduler-3359497473-5plhf           1/1    Running   0         34m              
 kube-system   kube-scheduler-3359497473-r7zg7           1/1    Running   1         34m              
 kube-system   pod-checkpointer-4kxtl                    1/1    Running   0         34m              
-kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0         33m
+kube-system   pod-checkpointer-4kxtl-ip-10-0-3-155      1/1    Running   0         33m
 ```
 
 ## Going Further
@@ -223,8 +224,8 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 |:-----|:------------|:--------|:--------|
 | controller_count | Number of controllers (i.e. masters) | 1 | 1 |
 | worker_count | Number of workers | 1 | 3 |
-| controller_type | EC2 instance type for controllers | "t2.small" | See below |
-| worker_type | EC2 instance type for workers | "t2.small" | See below |
+| controller_type | EC2 instance type for controllers | "t3.small" | See below |
+| worker_type | EC2 instance type for workers | "t3.small" | See below |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | "0" (i.e. auto) | "400" |

docs/atomic/bare-metal.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.12.2 cluster on bare-metal with Fedora Atomic.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.3 cluster on bare-metal with Fedora Atomic.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora Atomic via kickstart, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via cloud-init.
 
@@ -235,7 +235,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-atomic/kubernete
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.13.3"
   
   providers = {
     local = "local.default"
@@ -360,10 +360,10 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
-NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.12.2
-node2.example.com   Ready     11m       v1.12.2
-node3.example.com   Ready     11m       v1.12.2
+NAME                STATUS  ROLES              AGE  VERSION
+node1.example.com   Ready   controller,master  10m  v1.13.3
+node2.example.com   Ready   node               10m  v1.13.3
+node3.example.com   Ready   node               10m  v1.13.3
 ```
 
 List the pods.
@@ -374,6 +374,7 @@ NAMESPACE     NAME                                       READY     STATUS    RES
 kube-system   calico-node-6qp7f                          2/2       Running   1          11m
 kube-system   calico-node-gnjrm                          2/2       Running   0          11m
 kube-system   calico-node-llbgt                          2/2       Running   0          11m
+kube-system   coredns-1187388186-dj3pd                   1/1       Running   0          11m
 kube-system   coredns-1187388186-mx9rt                   1/1       Running   0          11m
 kube-system   kube-apiserver-7336w                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-b9chx   1/1       Running   0          11m

docs/atomic/digital-ocean.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on DigitalOcean with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on DigitalOcean with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -77,7 +77,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-atomic/kubern
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.13.3"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -151,10 +151,10 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.12.2
-10.132.115.81    Ready     10m       v1.12.2
-10.132.124.107   Ready     10m       v1.12.2
+NAME               STATUS  ROLES              AGE  VERSION
+nemo-controller-0  Ready   controller,master  10m  v1.13.3
+nemo-worker-0      Ready   node               10m  v1.13.3
+nemo-worker-1      Ready   node               10m  v1.13.3
 ```
 
 List the pods.
@@ -162,19 +162,20 @@ List the pods.
 ```
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
+kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
+kube-system   flannel-1cq1v                              2/2       Running   0          11m
+kube-system   flannel-hq9t0                              2/2       Running   1          11m
+kube-system   flannel-v0g9w                              2/2       Running   0          11m
 kube-system   kube-apiserver-n10qr                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-37gtw   1/1       Running   1          11m
 kube-system   kube-controller-manager-3271970485-p52t5   1/1       Running   0          11m
-kube-system   kube-flannel-1cq1v                         2/2       Running   0          11m
-kube-system   kube-flannel-hq9t0                         2/2       Running   1          11m
-kube-system   kube-flannel-v0g9w                         2/2       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m
 kube-system   kube-proxy-fh3td                           1/1       Running   0          11m
 kube-system   kube-proxy-k35rc                           1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-2bc4c            1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-b7q47            1/1       Running   1          11m
 kube-system   pod-checkpointer-pr1lq                     1/1       Running   0          11m
-kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0          10m
+kube-system   pod-checkpointer-pr1lq-nemo-controller-0   1/1       Running   0          10m
 ```
 
 ## Going Further
@@ -203,9 +204,9 @@ Clusters create DNS A records `${cluster_name}.${dns_zone}` to resolve to contro
 You'll need a registered domain name or delegated subdomain in Digital Ocean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.
 
 ```tf
+# Declare a DigitalOcean record to also create a zone file
 resource "digitalocean_domain" "zone-for-clusters" {
   name       = "do.example.com"
-  # Digital Ocean oddly requires an IP here. You may have to delete the A record it makes. :(
   ip_address = "8.8.8.8"
 }
 ```

docs/atomic/google-cloud.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Fedora does not publish official images for Google Cloud so you must prepare them yourself. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on Google Compute Engine with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on Google Compute Engine with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -121,7 +121,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-atomic/kuberne
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.13.3"
   
   providers = {
     google   = "google.default"
@@ -196,10 +196,10 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
 ```
 
 List the pods.
@@ -210,6 +210,7 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m

docs/cl/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on AWS with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.
 
@@ -18,23 +18,15 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
 
 ```sh
 $ terraform version
-Terraform v0.11.7
+Terraform v0.11.11
 ```
 
-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -64,6 +56,10 @@ provider "aws" {
   shared_credentials_file = "/home/user/.config/aws/credentials"
 }
 
+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
   version = "~> 1.0"
   alias = "default"
@@ -96,7 +92,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.13.3"
 
   providers = {
     aws = "aws.default"
@@ -117,7 +113,7 @@ module "aws-tempest" {
 
   # optional
   worker_count = 2
-  worker_type  = "t2.medium"
+  worker_type  = "t3.small"
 }
 ```
 
@@ -168,10 +164,10 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.12.2
-ip-10-0-19-112   Ready     34m       v1.12.2
-ip-10-0-4-22     Ready     34m       v1.12.2
+NAME           STATUS  ROLES              AGE  VERSION
+ip-10-0-3-155  Ready   controller,master  10m  v1.13.3
+ip-10-0-26-65  Ready   node               10m  v1.13.3
+ip-10-0-41-21  Ready   node               10m  v1.13.3
 ```
 
 List the pods.
@@ -183,6 +179,7 @@ kube-system   calico-node-1m5bf                         2/2    Running   0
 kube-system   calico-node-7jmr1                         2/2    Running   0         34m              
 kube-system   calico-node-bknc8                         2/2    Running   0         34m              
 kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m              
+kube-system   coredns-1187388186-qjnvp                  1/1    Running   0         34m
 kube-system   kube-apiserver-4mjbk                      1/1    Running   0         34m              
 kube-system   kube-controller-manager-3597210155-j2jbt  1/1    Running   1         34m              
 kube-system   kube-controller-manager-3597210155-j7g7x  1/1    Running   0         34m              
@@ -192,12 +189,12 @@ kube-system   kube-proxy-sbbsh                          1/1    Running   0
 kube-system   kube-scheduler-3359497473-5plhf           1/1    Running   0         34m              
 kube-system   kube-scheduler-3359497473-r7zg7           1/1    Running   1         34m              
 kube-system   pod-checkpointer-4kxtl                    1/1    Running   0         34m              
-kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0         33m
+kube-system   pod-checkpointer-4kxtl-ip-10-0-3-155      1/1    Running   0         33m
 ```
 
 ## Going Further
 
-Learn about [maintenance](/topics/maintenance) and [addons](/addons/overview).
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
 
 !!! note
     On Container Linux clusters, install the `CLUO` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@@ -239,8 +236,8 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 |:-----|:------------|:--------|:--------|
 | controller_count | Number of controllers (i.e. masters) | 1 | 1 |
 | worker_count | Number of workers | 1 | 3 |
-| controller_type | EC2 instance type for controllers | "t2.small" | See below |
-| worker_type | EC2 instance type for workers | "t2.small" | See below |
+| controller_type | EC2 instance type for controllers | "t3.small" | See below |
+| worker_type | EC2 instance type for workers | "t3.small" | See below |
 | os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
@@ -262,3 +259,8 @@ Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-typ
 
 !!! tip "MTU"
     If your EC2 instance type supports [Jumbo frames](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) (most do), we recommend you change the `network_mtu` to 8981! You will get better pod-to-pod bandwidth.
+
+#### Spot
+
+Add `worker_price = "0.10"` to use spot instance workers (instead of "on-demand") and set a maximum spot price in USD. Clusters can tolerate spot market interuptions fairly well (reschedules pods, but cannot drain) to save money, with the tradeoff that requests for workers may go unfulfilled.
+

docs/cl/azure.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on Azure with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on Azure with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.
 
@@ -21,23 +21,15 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
 
 ```sh
 $ terraform version
-Terraform v0.11.7
+Terraform v0.11.11
 ```
 
-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -62,6 +54,10 @@ provider "azurerm" {
   alias   = "default"
 }
 
+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
   version = "~> 1.0"
   alias   = "default"
@@ -91,7 +87,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.
 
 ```tf
 module "azure-ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.13.3"
 
   providers = {
     azurerm  = "azurerm.default"
@@ -112,7 +108,7 @@ module "azure-ramius" {
   asset_dir          = "/home/user/.secrets/clusters/ramius"
 
   # optional
-  worker_count    = 3
+  worker_count    = 2
   host_cidr       = "10.0.0.0/20"
 }
 ```
@@ -165,10 +161,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/ramius/auth/kubeconfig
 $ kubectl get nodes
 NAME                  STATUS  ROLES              AGE  VERSION
-ramius-controller-0   Ready   controller,master  24m  v1.12.2
-ramius-worker-000001  Ready   node               25m  v1.12.2
-ramius-worker-000002  Ready   node               24m  v1.12.2
-ramius-worker-000005  Ready   node               24m  v1.12.2
+ramius-controller-0   Ready   controller,master  24m  v1.13.3
+ramius-worker-000001  Ready   node               25m  v1.13.3
+ramius-worker-000002  Ready   node               24m  v1.13.3
 ```
 
 List the pods.
@@ -177,17 +172,16 @@ List the pods.
 $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                        READY  STATUS    RESTARTS  AGE
 kube-system   coredns-7c6fbb4f4b-b6qzx                    1/1    Running   0         26m
+kube-system   coredns-7c6fbb4f4b-j2k3d                    1/1    Running   0         26m
+kube-system   flannel-bwf24                               2/2    Running   2         26m
+kube-system   flannel-ks5qb                               2/2    Running   0         26m
+kube-system   flannel-tq2wg                               2/2    Running   0         26m
 kube-system   kube-apiserver-hxgsx                        1/1    Running   3         26m
 kube-system   kube-controller-manager-5ff9cd7bb6-b942n    1/1    Running   0         26m
 kube-system   kube-controller-manager-5ff9cd7bb6-bbr6w    1/1    Running   0         26m
-kube-system   kube-flannel-bwf24                          2/2    Running   2         26m
-kube-system   kube-flannel-ks5qb                          2/2    Running   0         26m
-kube-system   kube-flannel-nghsx                          2/2    Running   2         26m
-kube-system   kube-flannel-tq2wg                          2/2    Running   0         26m
 kube-system   kube-proxy-j4vpq                            1/1    Running   0         26m
 kube-system   kube-proxy-jxr5d                            1/1    Running   0         26m
 kube-system   kube-proxy-lbdw5                            1/1    Running   0         26m
-kube-system   kube-proxy-v8r7c                            1/1    Running   0         26m
 kube-system   kube-scheduler-5f76d69686-s4fbx             1/1    Running   0         26m
 kube-system   kube-scheduler-5f76d69686-vgdgn             1/1    Running   0         26m
 kube-system   pod-checkpointer-cnqdg                      1/1    Running   0         26m
@@ -255,7 +249,7 @@ Reference the DNS zone with `"${azurerm_dns_zone.clusters.name}"` and its resour
 | controller_type | Machine type for controllers | "Standard_DS1_v2" | See below |
 | worker_type | Machine type for workers | "Standard_F1" | See below |
 | os_image | Channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha |
-| disk_size | Size of the disk GB | "40" | "100" |
+| disk_size | Size of the disk in GB | "40" | "100" |
 | worker_priority | Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Low |
 | controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |

docs/cl/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.12.2 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.13.3 cluster on bare-metal with Container Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -113,28 +113,20 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system.
+Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
 wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
 tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
-sudo mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox /usr/local/bin/
+mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.2
 ```
 
-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  matchbox = "/usr/local/bin/terraform-provider-matchbox"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -149,12 +141,17 @@ Configure the Matchbox provider to use your Matchbox API endpoint and client cer
 
 ```tf
 provider "matchbox" {
+  version = "0.2.2"
   endpoint    = "matchbox.example.com:8081"
   client_cert = "${file("~/.config/matchbox/client.crt")}"
   client_key  = "${file("~/.config/matchbox/client.key")}"
   ca          = "${file("~/.config/matchbox/ca.crt")}"
 }
 
+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
   version = "~> 1.0"
   alias = "default"
@@ -182,7 +179,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.3"
   
   providers = {
     local = "local.default"
@@ -291,9 +288,9 @@ Apply complete! Resources: 55 added, 0 changed, 0 destroyed.
 To watch the install to disk (until machines reboot from disk), SSH to port 2222.
 
 ```
-# before v1.12.2
+# before v1.13.3
 $ ssh debug@node1.example.com
-# after v1.12.2
+# after v1.13.3
 $ ssh -p 2222 core@node1.example.com
 ```
 
@@ -317,10 +314,10 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
-NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.12.2
-node2.example.com   Ready     11m       v1.12.2
-node3.example.com   Ready     11m       v1.12.2
+NAME                STATUS  ROLES              AGE  VERSION
+node1.example.com   Ready   controller,master  10m  v1.13.3
+node2.example.com   Ready   node               10m  v1.13.3
+node3.example.com   Ready   node               10m  v1.13.3
 ```
 
 List the pods.
@@ -331,6 +328,7 @@ NAMESPACE     NAME                                       READY     STATUS    RES
 kube-system   calico-node-6qp7f                          2/2       Running   1          11m
 kube-system   calico-node-gnjrm                          2/2       Running   0          11m
 kube-system   calico-node-llbgt                          2/2       Running   0          11m
+kube-system   coredns-1187388186-dj3pd                   1/1       Running   0          11m
 kube-system   coredns-1187388186-mx9rt                   1/1       Running   0          11m
 kube-system   kube-apiserver-7336w                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-b9chx   1/1       Running   0          11m

docs/cl/digital-ocean.md

@@ -1,6 +1,6 @@
 # Digital Ocean
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on DigitalOcean with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -18,23 +18,15 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your sys
 
 ```sh
 $ terraform version
-Terraform v0.11.7
+Terraform v0.11.11
 ```
 
-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -63,6 +55,10 @@ provider "digitalocean" {
   alias = "default"
 }
 
+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
   version = "~> 1.0"
   alias = "default"
@@ -90,7 +86,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.13.3"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -163,10 +159,10 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
-NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.12.2
-10.132.115.81    Ready     10m       v1.12.2
-10.132.124.107   Ready     10m       v1.12.2
+NAME               STATUS  ROLES              AGE  VERSION
+nemo-controller-0  Ready   controller,master  10m  v1.13.3
+nemo-worker-0      Ready   node               10m  v1.13.3
+nemo-worker-1      Ready   node               10m  v1.13.3
 ```
 
 List the pods.
@@ -174,19 +170,20 @@ List the pods.
 ```
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
+kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
+kube-system   flannel-1cq1v                              2/2       Running   0          11m
+kube-system   flannel-hq9t0                              2/2       Running   1          11m
+kube-system   flannel-v0g9w                              2/2       Running   0          11m
 kube-system   kube-apiserver-n10qr                       1/1       Running   0          11m
 kube-system   kube-controller-manager-3271970485-37gtw   1/1       Running   1          11m
 kube-system   kube-controller-manager-3271970485-p52t5   1/1       Running   0          11m
-kube-system   kube-flannel-1cq1v                         2/2       Running   0          11m
-kube-system   kube-flannel-hq9t0                         2/2       Running   1          11m
-kube-system   kube-flannel-v0g9w                         2/2       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m
 kube-system   kube-proxy-fh3td                           1/1       Running   0          11m
 kube-system   kube-proxy-k35rc                           1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-2bc4c            1/1       Running   0          11m
 kube-system   kube-scheduler-3895335239-b7q47            1/1       Running   1          11m
 kube-system   pod-checkpointer-pr1lq                     1/1       Running   0          11m
-kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0          10m
+kube-system   pod-checkpointer-pr1lq-nemo-controller-0   1/1       Running   0          10m
 ```
 
 ## Going Further
@@ -217,9 +214,9 @@ Clusters create DNS A records `${cluster_name}.${dns_zone}` to resolve to contro
 You'll need a registered domain name or delegated subdomain in Digital Ocean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.
 
 ```tf
+# Declare a DigitalOcean record to also create a zone file
 resource "digitalocean_domain" "zone-for-clusters" {
   name       = "do.example.com"
-  # Digital Ocean oddly requires an IP here. You may have to delete the A record it makes. :(
   ip_address = "8.8.8.8"
 }
 ```

docs/cl/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.12.2 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.13.3 cluster on Google Compute Engine with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -21,20 +21,12 @@ $ terraform version
 Terraform v0.11.7
 ```
 
-Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
 
 ```sh
-wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
-sudo mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct /usr/local/bin/
-```
-
-Add the plugin to your `~/.terraformrc`.
-
-```
-providers {
-  ct = "/usr/local/bin/terraform-provider-ct"
-}
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
 ```
 
 Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
@@ -65,6 +57,10 @@ provider "google" {
   region      = "us-central1"
 }
 
+provider "ct" {
+  version = "0.3.0"
+}
+
 provider "local" {
   version = "~> 1.0"
   alias = "default"
@@ -97,7 +93,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
   
   providers = {
     google   = "google.default"
@@ -171,10 +167,10 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
 ```
 
 List the pods.
@@ -185,6 +181,7 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m

docs/index.md

@@ -11,29 +11,32 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](addons/overview/)
 
 ## Modules
 
-Typhoon provides a Terraform Module for each supported operating system and platform.
+Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.
 
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Container Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](atomic/aws.md) | alpha |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](atomic/bare-metal.md) | alpha |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](atomic/digital-ocean.md) | alpha |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |
-| Google Cloud  | Fedora Atomic    | [google-cloud/container-linux/kubernetes](atomic/google-cloud.md) | alpha |
 
-The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
+Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](atomic/aws.md) | alpha |
+| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](atomic/bare-metal.md) | alpha |
+| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](atomic/digital-ocean.md) | alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](atomic/google-cloud.md) | alpha |
 
 ## Documentation
 
@@ -46,7 +49,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
   
   providers = {
     google   = "google.default"
@@ -86,10 +89,10 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 ```
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.2
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
 ```
 
 List the pods.
@@ -100,6 +103,7 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
@@ -110,6 +114,7 @@ kube-system   kube-proxy-njn47                          1/1    Running   0
 kube-system   kube-scheduler-3895335239-5x87r           1/1    Running   0         6m
 kube-system   kube-scheduler-3895335239-bzrrt           1/1    Running   1         6m
 kube-system   pod-checkpointer-l6lrt                    1/1    Running   0         6m
+kube-system   pod-checkpointer-l6lrt-controller-0       1/1    Running   0         6m
 ```
 
 ## Help

docs/topics/maintenance.md

@@ -18,7 +18,7 @@ module "google-cloud-yavin" {
 }
 
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.12.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.13.3"
   ...
 }
 ```
@@ -126,3 +126,147 @@ Typhoon supports multi-controller clusters, so it is possible to upgrade a clust
 
 !!! warning
     Typhoon does not support or document node replacement as an upgrade strategy. It limits Typhoon's ability to make infrastructure and architectural changes between tagged releases. 
+
+### Terraform Plugins Directory
+
+Use the Terraform 3rd-party [plugin directory](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) `~/.terraform.d/plugins` to keep versioned copies of the `terraform-provider-ct` and `terraform-provider-matchbox` plugins. The plugin directory replaces the `~/.terraformrc` file to allow 3rd party plugins to be defined and versioned independently (rather than globally).
+
+```
+# ~/.terraformrc (DEPRECATED)
+providers {
+  ct = "/usr/local/bin/terraform-provider-ct"
+  matchbox = "/usr/local/bin/terraform-provider-matchbox"
+}
+```
+
+Migrate to using the Terraform plugin directory. Move `~/.terraformrc` to a backup location.
+
+```
+mv ~/.terraformrc ~/.terraform-backup
+```
+
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`. Download the **same version** of `terraform-provider-ct` you were using with `~/.terraformrc`, updating only be done as a followup and is **only** safe for v1.12.2+ clusters!
+
+```sh
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.2.1/terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.2.1-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.2.1-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.2.1
+```
+
+If you use bare-metal, add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the versioned name.
+
+```sh
+wget https://github.com/coreos/terraform-provider-matchbox/releases/download/v0.2.2/terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
+tar xzf terraform-provider-matchbox-v0.2.2-linux-amd64.tar.gz
+mv terraform-provider-matchbox-v0.2.2-linux-amd64/terraform-provider-matchbox ~/.terraform.d/plugins/terraform-provider-matchbox_v0.2.2
+```
+
+Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
+
+```
+$ tree ~/.terraform.d/
+/home/user/.terraform.d/
+└── plugins
+    ├── terraform-provider-ct_v0.2.1
+    └── terraform-provider-matchbox_v0.2.2
+```
+
+In each Terraform working directory, set the version of each provider.
+
+```
+# providers.tf
+
+provider "matchbox" {
+  version = "0.2.2"
+  ...
+}
+
+provider "ct" {
+  version = "0.2.1"
+}
+```
+
+Run `terraform init` to ensure plugin version requirements are met. Verify `terraform plan` does not produce a diff, since the plugin versions should be the same as previously.
+
+```
+$ terraform init
+$ terraform plan
+```
+
+### Upgrade terraform-provider-ct
+
+The [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin parses, validates, and converts Container Linux Configs into Ignition user-data for provisioning instances. Previously, updating the plugin re-provisioned controller nodes and was destructive to clusters. With Typhoon v1.12.2+, the plugin can be updated in-place and on apply, only workers will be replaced.
+
+First, [migrate](#terraform-plugins-directory) to the Terraform 3rd-party plugin directory to allow 3rd-party plugins to be defined and versioned independently (rather than globally).
+
+Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
+
+```sh
+wget https://github.com/coreos/terraform-provider-ct/releases/download/v0.3.0/terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.3.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.3.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.3.0
+```
+
+Binary names are versioned. This enables the ability to upgrade different plugins and have clusters pin different versions.
+
+```
+$ tree ~/.terraform.d/
+/home/user/.terraform.d/
+└── plugins
+    ├── terraform-provider-ct_v0.2.1
+    ├── terraform-provider-ct_v0.3.0
+    └── terraform-provider-matchbox_v0.2.2
+```
+
+
+Update the version of the `ct` plugin in each Terraform working directory. Typhoon clusters managed in the working directory **must** be v1.12.2 or higher.
+
+```
+# providers.tf
+provider "ct" {
+  version = "0.3.0"
+}
+```
+
+Run init and plan to check that no diff is proposed for the controller nodes (a diff would destroy cluster state).
+
+```
+terraform init
+terraform plan
+```
+
+Apply the change. Worker nodes' user-data will be changed and workers will be replaced. Rollout happens slightly differently on each platform:
+
+
+#### AWS
+
+AWS creates a new worker ASG, then removes the old ASG. New workers join the cluster and old workers disappear. `terraform apply` will hang during this process.
+
+#### Azure
+
+Azure edits the worker scale set in-place instantly. Manually terminate workers to create replacement workers using the new user-data.
+
+#### Bare-Metal
+
+No action is needed. Bare-Metal machines do not re-PXE unless explicitly made to do so.
+
+#### DigitalOcean
+
+DigitalOcean destroys existing worker nodes and DNS records, then creates new workers and DNS records. DigitalOcean lacks a "managed group" notion. For worker droplets to join the cluster, you **must** taint the secret copying step to indicate it must be repeated to add the kubeconfig to new workers.
+
+```
+# old workers destroyed, new workers created
+terraform apply
+
+# add kubeconfig to new workers
+terraform state list | grep null_resource
+terraform taint -module digital-ocean-nemo null_resource.copy-worker-secrets.N
+terraform apply
+```
+
+Expect downtime.
+
+#### Google Cloud
+
+Google Cloud creates a new worker template and edits the worker instance group instantly. Manually terminate workers and replacement workers will use the user-data.
+

docs/topics/performance.md

@@ -6,7 +6,7 @@ Provisioning times vary based on the operating system and platform. Sampling the
 
 | Platform      | Apply | Destroy |
 |---------------|-------|---------|
-| AWS           | 6 min | 5 min   |
+| AWS           | 6 min | 4 min   |
 | Azure         | 7 min | 7 min   |
 | Bare-Metal    | 10-15 min | NA  |
 | Digital Ocean | 3 min 30 sec | 20 sec |
@@ -24,17 +24,17 @@ Network performance varies based on the platform and CNI plugin. `iperf` was use
 
 | Platform / Plugin          | Theory | Host to Host | Pod to Pod   |
 |----------------------------|-------:|-------------:|-------------:|
-| AWS (flannel)              | ?      | 976 MB/s     | 900-999 MB/s |
-| AWS (calico, MTU 1480)     | ?      | 976 MB/s     | 100-350 MB/s |
-| AWS (calico, MTU 8981)     | ?      | 976 MB/s     | 900-999 MB/s |
-| Azure (flannel)            | ?      | 749 MB/s     | 680 MB/s     |
-| Bare-Metal (flannel)       | 1 GB/s | ~940 MB/s    | 903 MB/s     | 
-| Bare-Metal (calico)        | 1 GB/s | ~940 MB/s    | 931 MB/s     |
-| Bare-Metal (flannel, bond) | 3 GB/s |  2.3 GB/s    | 1.17 GB/s    | 
-| Bare-Metal (calico, bond)  | 3 GB/s |  2.3 GB/s    | 1.17 GB/s    |
-| Digital Ocean              | ?      | ~940 MB/s     | 820-880 MB/s |
-| Google Cloud (flannel)     | ?      | 1.94 GB/s    | 1.76 GB/s    |
-| Google Cloud (calico)      | ?      | 1.94 GB/s    | 1.81 GB/s    |
+| AWS (flannel)              | 5 Gb/s | 4.94 Gb/s    | 4.89 Gb/s    |
+| AWS (calico, MTU 1480)     | 5 Gb/s | 4.94 Gb/s    | 4.42 Gb/s    |
+| AWS (calico, MTU 8981)     | 5 Gb/s | 4.94 Gb/s    | 4.75 Gb/s    |
+| Azure (flannel)            | Varies |  749 Mb/s    | 680 Mb/s     |
+| Bare-Metal (flannel)       | 1 Gb/s |  940 Mb/s    | 903 Mb/s     |
+| Bare-Metal (calico)        | 1 Gb/s |  940 Mb/s    | 931 Mb/s     |
+| Bare-Metal (flannel, bond) | 3 Gb/s |  2.3 Gb/s    | 1.17 Gb/s    |
+| Bare-Metal (calico, bond)  | 3 Gb/s |  2.3 Gb/s    | 1.17 Gb/s    |
+| Digital Ocean              | 2 Gb/s | 1.97 Gb/s    | 1.64 Gb/s    |
+| Google Cloud (flannel)     | 2 Gb/s | 1.94 Gb/s    | 1.76 Gb/s    |
+| Google Cloud (calico)      | 2 Gb/s | 1.94 Gb/s    | 1.81 Gb/s    |
 
 Notes: