Add CHANGES note about reducing the pod eviciton timeout

* https://github.com/prometheus/prometheus/releases/tag/v2.7.1

.github/ISSUE_TEMPLATE.md

@@ -5,8 +5,8 @@
 ### Environment
 
 * Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, fedora-atomic
-* Ref: Release version or Git SHA (reporting latest is **not** helpful)
+* OS: container-linux, flatcar-linux, or fedora-atomic
+* Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
 * Terraform: `terraform version` (reporting latest is **not** helpful)
 * Plugins: Provider plugin versions (reporting latest is **not** helpful)
 

CHANGES.md

@@ -4,20 +4,177 @@ Notable changes between versions.
 
 ## Latest
 
-* Kubernetes [v1.12.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1121)
-* Update etcd from v3.3.9 to [v3.3.10](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10)
-* Update CoreDNS from 1.1.3 to 1.2.2
-* Update Calico from v3.2.1 to v3.2.3
-* On multi-controller clusters, raise scheduler and controller-manager replics to equal the number of controller nodes ([#312](https://github.com/poseidon/typhoon/pull/312))
-  * Single-controller clusters continue to run 2 replicas as before
-* Raise default CoreDNS replica count to the larger of 2 or the number of controller nodes ([#313](https://github.com/poseidon/typhoon/pull/313))
-  * Add AntiAffinity preferred rule to favor spreading CoreDNS pods
-* Annotate Kubernetes control plane and addons to start containers with the Docker runtime's default seccomp profile ([#319](https://github.com/poseidon/typhoon/pull/319))
-  * Override Kubernetes default behavior that starts containers with seccomp=unconfined
+## v1.13.3
+
+* Kubernetes [v1.13.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1133)
+* Update etcd from v3.3.10 to [v3.3.11](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3311-2019-1-11)
+* Update CoreDNS from v1.3.0 to [v1.3.1](https://coredns.io/2019/01/13/coredns-1.3.1-release/)
+  * Switch from the `proxy` plugin to the faster `forward` plugin for upsteam resolvers
+* Update Calico from v3.4.0 to [v3.5.0](https://docs.projectcalico.org/v3.5/releases/)
+* Update flannel from v0.10.0 to [v0.11.0](https://github.com/coreos/flannel/releases/tag/v0.11.0)
+* Reduce pod eviction timeout for deleting pods on unready nodes to 1 minute
+  * Respond more quickly to node preemption (previously 5 minutes)
+* Fix automatic worker deletion on shutdown for cloud platforms
+  * Lowering Kubelet privileges in [#372](https://github.com/poseidon/typhoon/pull/372) dropped a needed node deletion authorization. Scale-in due to manual terraform apply (any cloud), AWS spot termination, or Azure low priority deletion left old nodes registered, requiring manual deletion (`kubectl delete node name`)
+
+#### AWS
+
+* Add `ingress_zone_id` output with the NLB DNS name's Route53 zone for use in alias records ([#380](https://github.com/poseidon/typhoon/pull/380))
 
 #### Azure
 
-* Remove admin_password field (disabled) since it is now optional
+* Fix azure provider warning, `public_ip` `allocation_method` replaces `public_ip_address_allocation`
+  * Require `terraform-provider-azurerm` v1.21+ (action required)
+
+#### Addons
+
+* Update nginx-ingress from v0.21.0 to v0.22.0
+* Update Prometheus from v2.6.0 to v2.7.1
+* Update kube-state-metrics from v1.4.0 to v1.5.0
+  * Fix ClusterRole to collect and export PodDisruptionBudget metrics ([#383](https://github.com/poseidon/typhoon/pull/383))
+* Update node-exporter from v0.15.2 to v0.17.0
+* Update Grafana from v5.4.2 to v5.4.3
+
+## v1.13.2
+
+* Kubernetes [v1.13.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1132)
+* Add ServiceAccounts for `kube-apiserver` and `kube-scheduler` ([#370](https://github.com/poseidon/typhoon/pull/370))
+* Use lower-privilege TLS client certificates for Kubelets ([#372](https://github.com/poseidon/typhoon/pull/372))
+* Use HTTPS liveness probes for `kube-scheduler` and `kube-controller-manager` ([#377](https://github.com/poseidon/typhoon/pull/377))
+* Update CoreDNS from v1.2.6 to [v1.3.0](https://coredns.io/2018/12/15/coredns-1.3.0-release/)
+* Allow the `certificates.k8s.io` API to issue certificates signed by the cluster CA ([#376](https://github.com/poseidon/typhoon/pull/376))
+  * Configure controller manager to sign CSRs that are manually [approved](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster) by an administrator
+
+#### AWS
+
+* Change `controller_type` and `worker_type` default from t2.small to t3.small ([#365](https://github.com/poseidon/typhoon/pull/365))
+  * t3.small is cheaper, provides 2 vCPU (instead of 1), and 5 Gbps of pod-to-pod bandwidth!
+
+#### Bare-Metal
+
+* Remove the `kubeconfig` output variable
+
+#### Addons
+
+* Update Prometheus from v2.5.0 to v2.6.0
+
+## v1.13.1
+
+* Kubernetes [v1.13.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1131)
+* Update Calico from v3.3.2 to [v3.4.0](https://docs.projectcalico.org/v3.4/releases/) ([#362](https://github.com/poseidon/typhoon/pull/362))
+  * Install CNI plugins with an init container rather than a sidecar
+  * Improve the `calico-node` ClusterRole
+* Recommend updating `terraform-provider-ct` plugin from v0.2.1 to v0.3.0 ([#363](https://github.com/poseidon/typhoon/pull/363))
+  * [Migration](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct) instructions for upgrading `terraform-provider-ct` in-place for v1.12.2+ clusters (**action required**)
+  * [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`
+  * Require Container Linux 1688.5.3 or newer
+
+#### Google Cloud
+
+* Increase TCP proxy apiserver backend service timeout from 1 minute to 5 minutes ([#361](https://github.com/poseidon/typhoon/pull/361))
+  * Align `port-forward` behavior closer to AWS/Azure (no timeout)
+
+#### Addons
+
+* Update Grafana from v5.4.0 to v5.4.2
+
+## v1.13.0
+
+* Kubernetes [v1.13.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1130)
+* Update Calico from v3.3.1 to [v3.3.2](https://docs.projectcalico.org/v3.3/releases/)
+
+#### Addons
+
+* Update Grafana from v5.3.4 to v5.4.0
+* Disable Grafana login form, since admin user can't be disabled ([#352](https://github.com/poseidon/typhoon/pull/352))
+  * Example manifests aim to provide a read-only dashboard view
+
+## v1.12.3
+
+* Kubernetes [v1.12.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1123)
+* Add `enable_reporting` variable (default "false") to provide upstreams with usage data ([#345](https://github.com/poseidon/typhoon/pull/345))
+* Change kube-apiserver `--kubelet-preferred-address-types` to InternalIP,ExternalIP,Hostname
+* Update Calico from v3.3.0 to [v3.3.1](https://docs.projectcalico.org/v3.3/releases/)
+  * Disable Felix usage reporting by default ([#345](https://github.com/poseidon/typhoon/pull/345))
+* Improve flannel manifests
+  * [Rename](https://github.com/poseidon/terraform-render-bootkube/commit/d045a8e6b8eccfbb9d69bb51953b5a93d23f67f7) `kube-flannel` DaemonSet to `flannel` and `kube-flannel-cfg` ConfigMap to `flannel-config` 
+  * [Drop](https://github.com/poseidon/terraform-render-bootkube/commit/39f9afb3360ec642e5b98457c8bd07eda35b6c96) unused mounts and add a CPU resource request
+* Update CoreDNS from v1.2.4 to [v1.2.6](https://coredns.io/2018/11/05/coredns-1.2.6-release/)
+  * Enable CoreDNS `loop` and `loadbalance` plugins ([#340](https://github.com/poseidon/typhoon/pull/340))
+* Fix pod-checkpointer log noise and checkpointable pods detection ([#346](https://github.com/poseidon/typhoon/pull/346))
+* Use kubernetes-incubator/bootkube v0.14.0
+* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-plugins-directory) switching from `~/.terraformrc` to the Terraform [third-party plugins](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins) directory `~/.terraform.d/plugins/`.
+  * Allows pinning `terraform-provider-ct` and `terraform-provider-matchbox` versions
+  * Improves safety of later plugin version migrations
+
+#### Azure
+
+* Use eviction policy `Delete` for `Low` priority virtual machine scale set workers ([#343](https://github.com/poseidon/typhoon/pull/343))
+  * Fix issue where Azure defaults to `Deallocate` eviction policy, which required manually restarting deallocated instances. `Delete` policy aligns Azure with AWS and GCP behavior.
+  * Require `terraform-provider-azurerm` v1.19+ (action required)
+
+#### Bare-Metal
+
+* Add Kubelet `/etc/iscsi` and `iscsadm` mounts on bare-metal for iSCSI ([#103](https://github.com/poseidon/typhoon/pull/103))
+
+#### Addons
+
+* Update nginx-ingress from v0.20.0 to v0.21.0
+* Update Prometheus from v2.4.3 to v2.5.0
+* Update Grafana from v5.3.2 to v5.3.4
+
+## v1.12.2
+
+* Kubernetes [v1.12.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1122)
+* Update CoreDNS from 1.2.2 to [1.2.4](https://github.com/coredns/coredns/releases/tag/v1.2.4)
+* Update Calico from v3.2.3 to [v3.3.0](https://docs.projectcalico.org/v3.3/releases/)
+* Disable Kubelet read-only port ([#324](https://github.com/poseidon/typhoon/pull/324))
+* Fix CoreDNS AntiAffinity spec to prefer spreading replicas
+* Ignore controller node user-data changes ([#335](https://github.com/poseidon/typhoon/pull/335))
+  * Once all managed clusters use v1.12.2, it is possible to update `terraform-provider-ct`
+
+#### AWS
+
+* Add `disk_iops` variable for EBS volume IOPS ([#314](https://github.com/poseidon/typhoon/pull/314))
+
+#### Azure
+
+* Use new `azurerm_network_interface_backend_address_pool_association` ([#332](https://github.com/poseidon/typhoon/pull/332))
+  * Require `terraform-provider-azurerm` v1.17+ (action required)
+* Add `primary` field to `ip_configuration` needed by v1.17+ ([#331](https://github.com/poseidon/typhoon/pull/331))
+
+#### DigitalOcean
+
+* Add AAAA DNS records resolving to worker nodes ([#333](https://github.com/poseidon/typhoon/pull/333))
+  * Hosting IPv6 apps requires editing nginx-ingress with `hostNetwork: true`
+
+#### Google Cloud
+
+* Add an IPv6 address and IPv6 forwarding rules for load balancing IPv6 Ingress ([#334](https://github.com/poseidon/typhoon/pull/334))
+  * Add `ingress_static_ipv6` output variable for use in AAAA DNS records
+  * Allow serving IPv6 applications via Kubernetes Ingress
+
+#### Addons
+
+* Configure Heapster to scrape Kubelets with bearer token auth ([#323](https://github.com/poseidon/typhoon/pull/323))
+* Update Grafana from v5.3.1 to v5.3.2
+
+## v1.12.1
+
+* Kubernetes [v1.12.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1121)
+* Update etcd from v3.3.9 to [v3.3.10](https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3310-2018-10-10)
+* Update CoreDNS from 1.1.3 to [1.2.2](https://github.com/coredns/coredns/releases/tag/v1.2.2)
+* Update Calico from v3.2.1 to [v3.2.3](https://docs.projectcalico.org/v3.2/releases/)
+* Raise scheduler and controller-manager replicas to the larger of 2 or the number of controller nodes ([#312](https://github.com/poseidon/typhoon/pull/312))
+  * Single-controller clusters continue to run 2 replicas as before
+* Raise default CoreDNS replicas to the larger of 2 or the number of controller nodes ([#313](https://github.com/poseidon/typhoon/pull/313))
+  * Add AntiAffinity preferred rule to favor spreading CoreDNS pods
+* Annotate control plane and addon containers to use the Docker runtime seccomp profile ([#319](https://github.com/poseidon/typhoon/pull/319))
+  * Override Kubernetes default behavior that starts containers with `seccomp=unconfined`
+
+#### Azure
+
+* Remove `admin_password` field (disabled) since it is now optional
   * Require `terraform-provider-azurerm` v1.16+ (action required)
 
 #### Bare-Metal

README.md

@@ -11,29 +11,32 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Modules
 
-Typhoon provides a Terraform Module for each supported operating system and platform.
+Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.
 
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
-| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
 
-The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
+Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
+| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
+| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
 
 ## Documentation
 
@@ -47,7 +50,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.3"
   
   providers = {
     google   = "google.default"
@@ -87,10 +90,10 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 ```sh
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
-NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.12.1
+NAME                                       ROLES              STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.3
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.3
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.3
 ```
 
 List the pods.
@@ -102,6 +105,7 @@ kube-system   calico-node-1cs8z                         2/2    Running   0
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
 kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
@@ -111,6 +115,7 @@ kube-system   kube-proxy-njn47                          1/1    Running   0
 kube-system   kube-scheduler-3895335239-5x87r           1/1    Running   0         6m
 kube-system   kube-scheduler-3895335239-bzrrt           1/1    Running   1         6m
 kube-system   pod-checkpointer-l6lrt                    1/1    Running   0         6m
+kube-system   pod-checkpointer-l6lrt-controller-0       1/1    Running   0         6m
 ```
 
 ## Non-Goals

addons/grafana/dashboards.yaml

@@ -1963,7 +1963,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_cpu{mode=\"idle\"}[2m])) * 100",
+                  "expr": "sum(rate(node_cpu_seconds_total{mode=\"idle\"}[2m])) * 100",
                   "hide": false,
                   "intervalFactor": 10,
                   "legendFormat": "",
@@ -2138,7 +2138,7 @@ data:
               "renderer": "flot",
               "seriesOverrides": [
                 {
-                  "alias": "node_memory_SwapFree{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
+                  "alias": "node_memory_SwapFree_bytes{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
                   "yaxis": 2
                 }
               ],
@@ -2148,7 +2148,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(node_memory_MemTotal) - sum(node_memory_MemFree) - sum(node_memory_Buffers) - sum(node_memory_Cached)",
+                  "expr": "sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes) - sum(node_memory_Buffers_bytes) - sum(node_memory_Cached_bytes)",
                   "intervalFactor": 2,
                   "legendFormat": "memory usage",
                   "metric": "memo",
@@ -2157,7 +2157,7 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(node_memory_Buffers)",
+                  "expr": "sum(node_memory_Buffers_bytes)",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory buffers",
@@ -2167,7 +2167,7 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(node_memory_Cached)",
+                  "expr": "sum(node_memory_Cached_bytes)",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory cached",
@@ -2177,7 +2177,7 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(node_memory_MemFree)",
+                  "expr": "sum(node_memory_MemFree_bytes)",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory free",
@@ -2268,7 +2268,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "((sum(node_memory_MemTotal) - sum(node_memory_MemFree) - sum(node_memory_Buffers) - sum(node_memory_Cached)) / sum(node_memory_MemTotal)) * 100",
+                  "expr": "((sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes) - sum(node_memory_Buffers_bytes) - sum(node_memory_Cached_bytes)) / sum(node_memory_MemTotal_bytes)) * 100",
                   "intervalFactor": 2,
                   "metric": "",
                   "refId": "A",
@@ -2355,7 +2355,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_disk_bytes_read[5m]))",
+                  "expr": "max(rate(node_disk_read_bytes_total[5m]))",
                   "hide": false,
                   "intervalFactor": 4,
                   "legendFormat": "read",
@@ -2364,14 +2364,14 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum(rate(node_disk_bytes_written[5m]))",
+                  "expr": "max(rate(node_disk_written_bytes_total[5m]))",
                   "intervalFactor": 4,
                   "legendFormat": "written",
                   "refId": "B",
                   "step": 20
                 },
                 {
-                  "expr": "sum(rate(node_disk_io_time_ms[5m]))",
+                  "expr": "max(rate(node_disk_io_time_seconds_total[5m]))",
                   "intervalFactor": 4,
                   "legendFormat": "io time",
                   "refId": "C",
@@ -2458,7 +2458,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "(sum(node_filesystem_size{device!=\"rootfs\"}) - sum(node_filesystem_free{device!=\"rootfs\"})) / sum(node_filesystem_size{device!=\"rootfs\"})",
+                  "expr": "(sum(node_filesystem_size_bytes{device!=\"rootfs\"}) - sum(node_filesystem_free_bytes{device!=\"rootfs\"})) / sum(node_filesystem_size_bytes{device!=\"rootfs\"})",
                   "intervalFactor": 2,
                   "refId": "A",
                   "step": 60,
@@ -2536,7 +2536,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_network_receive_bytes{device!~\"lo\"}[5m]))",
+                  "expr": "sum(rate(node_network_receive_bytes_total{device!~\"lo\"}[5m]))",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -2618,7 +2618,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum(rate(node_network_transmit_bytes{device!~\"lo\"}[5m]))",
+                  "expr": "sum(rate(node_network_transmit_bytes_total{device!~\"lo\"}[5m]))",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "",
@@ -4093,7 +4093,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "sum(100 - (avg by (instance) (rate(node_cpu{job=\"node-exporter\",mode=\"idle\"}[5m])) * 100)) / count(node_cpu{job=\"node-exporter\",mode=\"idle\"})",
+                  "expr": "sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}[5m])) * 100)) / count(node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "refId": "A",
@@ -4165,7 +4165,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "((sum(node_memory_MemTotal) - sum(node_memory_MemFree) - sum(node_memory_Buffers) - sum(node_memory_Cached)) / sum(node_memory_MemTotal)) * 100",
+                  "expr": "((sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes) - sum(node_memory_Buffers_bytes) - sum(node_memory_Cached_bytes)) / sum(node_memory_MemTotal_bytes)) * 100",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "refId": "A",
@@ -4237,7 +4237,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "(sum(node_filesystem_size{device!=\"rootfs\"}) - sum(node_filesystem_free{device!=\"rootfs\"})) / sum(node_filesystem_size{device!=\"rootfs\"})",
+                  "expr": "(sum(node_filesystem_size_bytes{device!=\"rootfs\"}) - sum(node_filesystem_free_bytes{device!=\"rootfs\"})) / sum(node_filesystem_size_bytes{device!=\"rootfs\"})",
                   "format": "time_series",
                   "intervalFactor": 2,
                   "refId": "A",
@@ -5476,7 +5476,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "100 - (avg by (cpu) (irate(node_cpu{mode=\"idle\", instance=\"$server\"}[5m])) * 100)",
+                  "expr": "100 - (avg by (cpu) (irate(node_cpu_seconds_total{mode=\"idle\", instance=\"$server\"}[5m])) * 100)",
                   "hide": false,
                   "intervalFactor": 10,
                   "legendFormat": "{{cpu}}",
@@ -5652,7 +5652,7 @@ data:
               "renderer": "flot",
               "seriesOverrides": [
                 {
-                  "alias": "node_memory_SwapFree{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
+                  "alias": "node_memory_SwapFree_bytes{instance=\"172.17.0.1:9100\",job=\"prometheus\"}",
                   "yaxis": 2
                 }
               ],
@@ -5662,7 +5662,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "node_memory_MemTotal{instance=\"$server\"} - node_memory_MemFree{instance=\"$server\"} - node_memory_Buffers{instance=\"$server\"} - node_memory_Cached{instance=\"$server\"}",
+                  "expr": "node_memory_MemTotal_bytes{instance=\"$server\"} - node_memory_MemFree_bytes{instance=\"$server\"} - node_memory_Buffers_bytes{instance=\"$server\"} - node_memory_Cached_bytes{instance=\"$server\"}",
                   "hide": false,
                   "interval": "",
                   "intervalFactor": 2,
@@ -5672,7 +5672,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "node_memory_Buffers{instance=\"$server\"}",
+                  "expr": "node_memory_Buffers_bytes{instance=\"$server\"}",
                   "interval": "",
                   "intervalFactor": 2,
                   "legendFormat": "memory buffers",
@@ -5689,7 +5689,7 @@ data:
                   "step": 10
                 },
                 {
-                  "expr": "node_memory_MemFree{instance=\"$server\"}",
+                  "expr": "node_memory_MemFree_bytes{instance=\"$server\"}",
                   "intervalFactor": 2,
                   "legendFormat": "memory free",
                   "metric": "",
@@ -5778,7 +5778,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "((node_memory_MemTotal{instance=\"$server\"} - node_memory_MemFree{instance=\"$server\"}  - node_memory_Buffers{instance=\"$server\"} - node_memory_Cached{instance=\"$server\"}) / node_memory_MemTotal{instance=\"$server\"}) * 100",
+                  "expr": "((node_memory_MemTotal_bytes{instance=\"$server\"} - node_memory_MemFree_bytes{instance=\"$server\"}  - node_memory_Buffers_bytes{instance=\"$server\"} - node_memory_Cached_bytes{instance=\"$server\"}) / node_memory_MemTotal_bytes{instance=\"$server\"}) * 100",
                   "intervalFactor": 2,
                   "refId": "A",
                   "step": 60,
@@ -5864,7 +5864,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "sum by (instance) (rate(node_disk_bytes_read{instance=\"$server\"}[2m]))",
+                  "expr": "sum by (instance) (rate(node_disk_read_bytes_total{instance=\"$server\"}[2m]))",
                   "hide": false,
                   "intervalFactor": 4,
                   "legendFormat": "read",
@@ -5873,14 +5873,14 @@ data:
                   "target": ""
                 },
                 {
-                  "expr": "sum by (instance) (rate(node_disk_bytes_written{instance=\"$server\"}[2m]))",
+                  "expr": "sum by (instance) (rate(node_disk_written_bytes_total{instance=\"$server\"}[2m]))",
                   "intervalFactor": 4,
                   "legendFormat": "written",
                   "refId": "B",
                   "step": 20
                 },
                 {
-                  "expr": "sum by (instance) (rate(node_disk_io_time_ms{instance=\"$server\"}[2m]))",
+                  "expr": "sum by (instance) (rate(node_disk_io_time_seconds_total{instance=\"$server\"}[2m]))",
                   "intervalFactor": 4,
                   "legendFormat": "io time",
                   "refId": "C",
@@ -5967,7 +5967,7 @@ data:
               },
               "targets": [
                 {
-                  "expr": "(sum(node_filesystem_size{device!=\"rootfs\",instance=\"$server\"}) - sum(node_filesystem_free{device!=\"rootfs\",instance=\"$server\"})) / sum(node_filesystem_size{device!=\"rootfs\",instance=\"$server\"})",
+                  "expr": "(sum(node_filesystem_size_bytes{device!=\"rootfs\",instance=\"$server\"}) - sum(node_filesystem_free_bytes{device!=\"rootfs\",instance=\"$server\"})) / sum(node_filesystem_size_bytes{device!=\"rootfs\",instance=\"$server\"})",
                   "intervalFactor": 2,
                   "refId": "A",
                   "step": 60,
@@ -6045,7 +6045,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_network_receive_bytes{instance=\"$server\",device!~\"lo\"}[5m])",
+                  "expr": "rate(node_network_receive_bytes_total{instance=\"$server\",device!~\"lo\"}[5m])",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "{{device}}",
@@ -6127,7 +6127,7 @@ data:
               "steppedLine": false,
               "targets": [
                 {
-                  "expr": "rate(node_network_transmit_bytes{instance=\"$server\",device!~\"lo\"}[5m])",
+                  "expr": "rate(node_network_transmit_bytes_total{instance=\"$server\",device!~\"lo\"}[5m])",
                   "hide": false,
                   "intervalFactor": 2,
                   "legendFormat": "{{device}}",
@@ -6184,7 +6184,7 @@ data:
             "multi": false,
             "name": "server",
             "options": [],
-            "query": "label_values(node_boot_time, instance)",
+            "query": "label_values(node_boot_time_seconds, instance)",
             "refresh": 1,
             "regex": "",
             "sort": 0,

addons/grafana/deployment.yaml

@@ -23,12 +23,14 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:5.3.1
+          image: grafana/grafana:5.4.3
           env:
             - name: GF_SERVER_HTTP_PORT
               value: "8080"
             - name: GF_AUTH_BASIC_ENABLED
               value: "false"
+            - name: GF_AUTH_DISABLE_LOGIN_FORM
+              value: "true"
             - name: GF_AUTH_ANONYMOUS_ENABLED
               value: "true"
             - name: GF_AUTH_ANONYMOUS_ORG_ROLE

addons/heapster/cluster-role-binding.yaml

@@ -5,7 +5,7 @@ metadata:
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
-  name: system:heapster
+  name: heapster
 subjects:
 - kind: ServiceAccount
   name: heapster

addons/heapster/cluster-role.yaml

@@ -0,0 +1,30 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: heapster
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - events
+  - namespaces
+  - nodes
+  - pods
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - deployments
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - nodes/stats
+  verbs:
+  - get

addons/heapster/deployment.yaml

@@ -23,7 +23,7 @@ spec:
           image: k8s.gcr.io/heapster-amd64:v1.5.4
           command:
             - /heapster
-            - --source=kubernetes.summary_api:''
+            - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
           livenessProbe:
             httpGet:
               path: /healthz

addons/nginx-ingress/aws/deployment.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/azure/deployment.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -22,7 +22,7 @@ spec:
     spec:
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -24,7 +24,7 @@ spec:
         node-role.kubernetes.io/node: ""
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend

addons/prometheus/config.yaml

@@ -102,7 +102,7 @@ data:
           regex: 'true'
         - action: labelmap
           regex: __meta_kubernetes_node_label_(.+)
-        - source_labels: [__meta_kubernetes_node_name]
+        - source_labels: [__meta_kubernetes_node_address_InternalIP]
           action: replace
           target_label: __address__
           replacement: '${1}:2381'

addons/prometheus/deployment.yaml

@@ -20,7 +20,7 @@ spec:
       serviceAccountName: prometheus
       containers:
         - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.4.3
+          image: quay.io/prometheus/prometheus:v2.7.1
           args:
             - --web.listen-address=0.0.0.0:9090
             - --config.file=/etc/prometheus/prometheus.yaml

addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml

@@ -3,7 +3,8 @@ kind: ClusterRole
 metadata:
   name: kube-state-metrics
 rules:
-- apiGroups: [""]
+- apiGroups:
+  - ""
   resources:
   - configmaps
   - secrets
@@ -17,23 +18,47 @@ rules:
   - persistentvolumes
   - namespaces
   - endpoints
-  verbs: ["list", "watch"]
-- apiGroups: ["extensions"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - extensions
   resources:
   - daemonsets
   - deployments
   - replicasets
-  verbs: ["list", "watch"]
-- apiGroups: ["apps"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - apps
   resources:
   - statefulsets
-  verbs: ["list", "watch"]
-- apiGroups: ["batch"]
+  - daemonsets
+  - deployments
+  - replicasets
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - batch
   resources:
   - cronjobs
   - jobs
-  verbs: ["list", "watch"]
-- apiGroups: ["autoscaling"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - autoscaling
   resources:
   - horizontalpodautoscalers
-  verbs: ["list", "watch"]
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - policy
+  resources:
+  - poddisruptionbudgets
+  verbs:
+  - list
+  - watch

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -24,7 +24,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.4.0
+        image: quay.io/coreos/kube-state-metrics:v1.5.0
         ports:
           - name: metrics
             containerPort: 8080
@@ -35,7 +35,7 @@ spec:
           initialDelaySeconds: 5
           timeoutSeconds: 5
       - name: addon-resizer
-        image: k8s.gcr.io/addon-resizer:1.7
+        image: k8s.gcr.io/addon-resizer:1.8.4
         resources:
           limits:
             cpu: 100m

addons/prometheus/exporters/kube-state-metrics/resizer-role-binding.yaml

@@ -6,7 +6,7 @@ metadata:
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role
-  name: kube-state-metrics-resizer
+  name: kube-state-metrics
 subjects:
 - kind: ServiceAccount
   name: kube-state-metrics

addons/prometheus/exporters/kube-state-metrics/resizer-role.yaml

@@ -1,15 +1,31 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
-  name: kube-state-metrics-resizer
+  name: kube-state-metrics
   namespace: monitoring
 rules:
-- apiGroups: [""]
+- apiGroups:
+  - ""
   resources:
   - pods
-  verbs: ["get"]
-- apiGroups: ["extensions"]
+  verbs:
+  - get
+- apiGroups:
+  - extensions
   resources:
   - deployments
-  resourceNames: ["kube-state-metrics"]
-  verbs: ["get", "update"]
+  resourceNames:
+  - kube-state-metrics
+  verbs:
+  - get
+  - update
+- apiGroups:
+  - apps
+  resources:
+  - deployments
+  resourceNames:
+  - kube-state-metrics
+  verbs:
+  - get
+  - update
+

addons/prometheus/exporters/node-exporter/daemonset.yaml

@@ -28,21 +28,24 @@ spec:
       hostPID: true
       containers:
       - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v0.15.2
+        image: quay.io/prometheus/node-exporter:v0.17.0
         args:
-          - "--path.procfs=/host/proc"
-          - "--path.sysfs=/host/sys"
+          - --path.procfs=/host/proc
+          - --path.sysfs=/host/sys
+          - --path.rootfs=/host/root
+          - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
+          - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
         ports:
           - name: metrics
             containerPort: 9100
             hostPort: 9100
         resources:
           requests:
-            memory: 30Mi
             cpu: 100m
-          limits:
             memory: 50Mi
+          limits:
             cpu: 200m
+            memory: 100Mi
         volumeMounts:
           - name: proc
             mountPath: /host/proc
@@ -50,6 +53,9 @@ spec:
           - name: sys
             mountPath: /host/sys
             readOnly: true
+          - name: root
+            mountPath: /host/root
+            readOnly: true
       tolerations:
         - effect: NoSchedule
           operator: Exists
@@ -60,3 +66,6 @@ spec:
         - name: sys
           hostPath:
             path: /sys
+        - name: root
+          hostPath:
+            path: /

addons/prometheus/rules.yaml

@@ -456,22 +456,22 @@ data:
     - name: node.rules
       rules:
       - record: instance:node_cpu:rate:sum
-        expr: sum(rate(node_cpu{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
+        expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
           BY (instance)
       - record: instance:node_filesystem_usage:sum
-        expr: sum((node_filesystem_size{mountpoint="/"} - node_filesystem_free{mountpoint="/"}))
+        expr: sum((node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"}))
           BY (instance)
       - record: instance:node_network_receive_bytes:rate:sum
-        expr: sum(rate(node_network_receive_bytes[3m])) BY (instance)
+        expr: sum(rate(node_network_receive_bytes_total[3m])) BY (instance)
       - record: instance:node_network_transmit_bytes:rate:sum
-        expr: sum(rate(node_network_transmit_bytes[3m])) BY (instance)
+        expr: sum(rate(node_network_transmit_bytes_total[3m])) BY (instance)
       - record: instance:node_cpu:ratio
-        expr: sum(rate(node_cpu{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
-          GROUP_LEFT() count(sum(node_cpu) BY (instance, cpu)) BY (instance)
+        expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
+          GROUP_LEFT() count(sum(node_cpu_seconds_total) BY (instance, cpu)) BY (instance)
       - record: cluster:node_cpu:sum_rate5m
-        expr: sum(rate(node_cpu{mode!="idle"}[5m]))
+        expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
       - record: cluster:node_cpu:ratio
-        expr: cluster:node_cpu:rate5m / count(sum(node_cpu) BY (instance, cpu))
+        expr: cluster:node_cpu:sum_rate5m / count(sum(node_cpu_seconds_total) BY (instance, cpu))
       - alert: NodeExporterDown
         expr: absent(up{job="node-exporter"} == 1)
         for: 10m
@@ -481,7 +481,7 @@ data:
           description: Prometheus could not scrape a node-exporter for more than 10m,
             or node-exporters have disappeared from discovery
       - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free[6h], 3600 * 24) < 0
+        expr: predict_linear(node_filesystem_free_bytes[6h], 3600 * 24) < 0
         for: 30m
         labels:
           severity: warning
@@ -489,7 +489,7 @@ data:
           description: device {{$labels.device}} on node {{$labels.instance}} is running
             full within the next 24 hours (mounted at {{$labels.mountpoint}})
       - alert: NodeDiskRunningFull
-        expr: predict_linear(node_filesystem_free[30m], 3600 * 2) < 0
+        expr: predict_linear(node_filesystem_free_bytes[30m], 3600 * 2) < 0
         for: 10m
         labels:
           severity: critical

aws/container-linux/kubernetes/README.md

@@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,4 +11,5 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -78,7 +78,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -88,6 +88,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -122,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -142,17 +143,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

aws/container-linux/kubernetes/controllers.tf

@@ -24,12 +24,13 @@ resource "aws_instance" "controllers" {
   instance_type = "${var.controller_type}"
 
   ami       = "${local.ami_id}"
-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
 
   # storage
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -38,12 +39,23 @@ resource "aws_instance" "controllers" {
   vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
 
   lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
   }
 }
 
-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
   count = "${var.controller_count}"
 
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@@ -56,10 +68,10 @@ data "template_file" "controller_config" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 
@@ -73,10 +85,3 @@ data "template_file" "etcds" {
     dns_zone     = "${var.dns_zone}"
   }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
-}

aws/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
@@ -5,6 +9,11 @@ output "ingress_dns_name" {
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_zone_id" {
+  value       = "${aws_lb.nlb.zone_id}"
+  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
+}
+
 # Outputs for worker pools
 
 output "vpc_id" {
@@ -23,7 +32,7 @@ output "worker_security_groups" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }
 
 # Outputs for custom load balancing

aws/container-linux/kubernetes/security.tf

@@ -104,27 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -300,27 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
   security_group_id = "${aws_security_group.worker.id}"
 

aws/container-linux/kubernetes/variables.tf

@@ -31,13 +31,13 @@ variable "worker_count" {
 
 variable "controller_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 
@@ -59,6 +59,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
   type        = "string"
   default     = ""
@@ -128,3 +134,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

aws/container-linux/kubernetes/workers.tf

@@ -13,7 +13,7 @@ module "workers" {
   spot_price      = "${var.worker_price}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -51,7 +51,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -60,6 +60,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -92,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -110,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.1 \
+            docker://k8s.gcr.io/hyperkube:v1.13.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/variables.tf

@@ -30,7 +30,7 @@ variable "count" {
 
 variable "instance_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type"
 }
 
@@ -52,6 +52,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
   type        = "string"
   default     = ""

aws/container-linux/kubernetes/workers/workers.tf

@@ -46,12 +46,13 @@ resource "aws_launch_configuration" "worker" {
   spot_price        = "${var.spot_price}"
   enable_monitoring = false
 
-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"
 
   # storage
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -64,20 +65,21 @@ resource "aws_launch_configuration" "worker" {
   }
 }
 
-# Worker Container Linux Config
-data "template_file" "worker_config" {
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
-
-  vars = {
-    kubeconfig            = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  }
-}
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
   pretty_print = false
   snippets     = ["${var.clc_snippets}"]
 }
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
+  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+
+  vars = {
+    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+  }
+}

aws/fedora-atomic/kubernetes/README.md

@@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,6 +11,7 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 
   # Fedora
   trusted_certs_dir = "/etc/pki/tls/certs"

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -19,24 +19,9 @@ write_files:
       ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
       ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
       ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/latest/meta-data/local-ipv4\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -55,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -65,6 +50,7 @@ write_files:
         --node-labels=node-role.kubernetes.io/master \
         --node-labels=node-role.kubernetes.io/controller="true" \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/kubernetes/kubeconfig
@@ -92,11 +78,10 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, cloud-metadata.service]
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/controllers.tf

@@ -30,6 +30,7 @@ resource "aws_instance" "controllers" {
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -38,7 +39,10 @@ resource "aws_instance" "controllers" {
   vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
 
   lifecycle {
-    ignore_changes = ["ami"]
+    ignore_changes = [
+      "ami",
+      "user_data",
+    ]
   }
 }
 
@@ -56,10 +60,10 @@ data "template_file" "controller-cloudinit" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(6, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(6, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

aws/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_dns_name" {
@@ -5,6 +9,11 @@ output "ingress_dns_name" {
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
+output "ingress_zone_id" {
+  value       = "${aws_lb.nlb.zone_id}"
+  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
+}
+
 # Outputs for worker pools
 
 output "vpc_id" {
@@ -23,7 +32,7 @@ output "worker_security_groups" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }
 
 # Outputs for custom load balancing

aws/fedora-atomic/kubernetes/security.tf

@@ -104,27 +104,6 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "controller-kubelet-read" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-read-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-bgp" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -300,27 +279,6 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
   self      = true
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "aws_security_group_rule" "worker-kubelet-read" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10255
-  to_port                  = 10255
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-kubelet-read-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10255
-  to_port   = 10255
-  self      = true
-}
-
 resource "aws_security_group_rule" "worker-bgp" {
   security_group_id = "${aws_security_group.worker.id}"
 

aws/fedora-atomic/kubernetes/variables.tf

@@ -31,13 +31,13 @@ variable "worker_count" {
 
 variable "controller_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for controllers"
 }
 
 variable "worker_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type for workers"
 }
 
@@ -53,6 +53,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (e.g. 100)"
+}
+
 variable "worker_price" {
   type        = "string"
   default     = ""
@@ -110,3 +116,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

aws/fedora-atomic/kubernetes/workers.tf

@@ -12,7 +12,7 @@ module "workers" {
   spot_price      = "${var.worker_price}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -1,23 +1,8 @@
 #cloud-config
 write_files:
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/latest/meta-data/local-ipv4\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -34,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -43,6 +28,7 @@ write_files:
         --network-plugin=cni \
         --node-labels=node-role.kubernetes.io/node \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/kubernetes/kubeconfig
     permissions: '0644'
@@ -68,8 +54,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -30,7 +30,7 @@ variable "count" {
 
 variable "instance_type" {
   type        = "string"
-  default     = "t2.small"
+  default     = "t3.small"
   description = "EC2 instance type"
 }
 
@@ -46,6 +46,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "disk_iops" {
+  type        = "string"
+  default     = "0"
+  description = "IOPS of the EBS volume (required for io1)"
+}
+
 variable "spot_price" {
   type        = "string"
   default     = ""

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -52,6 +52,7 @@ resource "aws_launch_configuration" "worker" {
   root_block_device {
     volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
+    iops        = "${var.disk_iops}"
   }
 
   # network
@@ -69,9 +70,9 @@ data "template_file" "worker-cloudinit" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(6, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(6, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

azure/container-linux/kubernetes/README.md

@@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

azure/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -10,4 +10,5 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -78,7 +78,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -88,6 +88,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -122,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -142,17 +143,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

azure/container-linux/kubernetes/controllers.tf

@@ -85,6 +85,7 @@ resource "azurerm_virtual_machine" "controllers" {
   lifecycle {
     ignore_changes = [
       "storage_os_disk",
+      "os_profile",
     ]
   }
 }
@@ -105,21 +106,25 @@ resource "azurerm_network_interface" "controllers" {
 
     # public IPv4
     public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
-
-    # backend address pool to which the NIC should be added
-    load_balancer_backend_address_pools_ids = ["${azurerm_lb_backend_address_pool.controller.id}"]
   }
 }
 
+# Add controller NICs to the controller backend address pool
+resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
+  network_interface_id    = "${azurerm_network_interface.controllers.id}"
+  ip_configuration_name   = "ip0"
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+}
+
 # Controller public IPv4 addresses
 resource "azurerm_public_ip" "controllers" {
   count               = "${var.controller_count}"
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-controller-${count.index}"
-  location                     = "${azurerm_resource_group.cluster.location}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-controller-${count.index}"
+  location          = "${azurerm_resource_group.cluster.location}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Controller Ignition configs
@@ -144,10 +149,10 @@ data "template_file" "controller-configs" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

azure/container-linux/kubernetes/lb.tf

@@ -17,20 +17,20 @@ resource "azurerm_dns_a_record" "apiserver" {
 resource "azurerm_public_ip" "apiserver-ipv4" {
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-apiserver-ipv4"
-  location                     = "${var.region}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-apiserver-ipv4"
+  location          = "${var.region}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Static IPv4 address for the ingress frontend
 resource "azurerm_public_ip" "ingress-ipv4" {
   resource_group_name = "${azurerm_resource_group.cluster.name}"
 
-  name                         = "${var.cluster_name}-ingress-ipv4"
-  location                     = "${var.region}"
-  sku                          = "Standard"
-  public_ip_address_allocation = "static"
+  name              = "${var.cluster_name}-ingress-ipv4"
+  location          = "${var.region}"
+  sku               = "Standard"
+  allocation_method = "Static"
 }
 
 # Network Load Balancer for apiservers and ingress

azure/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,7 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 # Outputs for Kubernetes Ingress
 
 output "ingress_static_ipv4" {
@@ -28,5 +32,5 @@ output "backend_address_pool_id" {
 }
 
 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+  value = "${module.bootkube.kubeconfig-kubelet}"
 }

azure/container-linux/kubernetes/require.tf

@@ -5,7 +5,7 @@ terraform {
 }
 
 provider "azurerm" {
-  version = "~> 1.16"
+  version = "~> 1.21"
 }
 
 provider "local" {
@@ -23,4 +23,3 @@ provider "template" {
 provider "tls" {
   version = "~> 1.0"
 }
-

azure/container-linux/kubernetes/security.tf

@@ -117,22 +117,6 @@ resource "azurerm_network_security_rule" "controller-kubelet" {
   destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "azurerm_network_security_rule" "controller-kubelet-read" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
-
-  name                        = "allow-kubelet-read"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
-  priority                    = "2035"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10255"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
-}
-
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
 
@@ -269,22 +253,6 @@ resource "azurerm_network_security_rule" "worker-kubelet" {
   destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
 }
 
-# Allow heapster / metrics-server to scrape kubelet read-only
-resource "azurerm_network_security_rule" "worker-kubelet-read" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
-
-  name                        = "allow-kubelet-read"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
-  priority                    = "2030"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10255"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
-}
-
 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
 

azure/container-linux/kubernetes/variables.tf

@@ -115,3 +115,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

azure/container-linux/kubernetes/workers.tf

@@ -15,7 +15,7 @@ module "workers" {
   priority = "${var.worker_priority}"
 
   # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig}"
+  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -51,7 +51,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -60,6 +60,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -92,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -110,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.1 \
+            docker://k8s.gcr.io/hyperkube:v1.13.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')

azure/container-linux/kubernetes/workers/ingress.tf

@@ -1 +0,0 @@
-

azure/container-linux/kubernetes/workers/workers.tf

@@ -37,7 +37,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
   os_profile {
     computer_name_prefix = "${var.name}-worker-"
     admin_username       = "core"
-    custom_data          = "${element(data.ct_config.worker-ignitions.*.rendered, count.index)}"
+    custom_data          = "${data.ct_config.worker-ignition.rendered}"
   }
 
   # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@@ -58,6 +58,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
 
     ip_configuration {
       name      = "ip0"
+      primary   = true
       subnet_id = "${var.subnet_id}"
 
       # backend address pool to which the NIC should be added
@@ -66,8 +67,9 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
   }
 
   # lifecycle
-  priority            = "${var.priority}"
   upgrade_policy_mode = "Manual"
+  priority            = "${var.priority}"
+  eviction_policy     = "Delete"
 }
 
 # Scale up or down to maintain desired number, tolerating deallocations.
@@ -93,20 +95,20 @@ resource "azurerm_autoscale_setting" "workers" {
 }
 
 # Worker Ignition configs
-data "ct_config" "worker-ignitions" {
-  content      = "${data.template_file.worker-configs.rendered}"
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
   pretty_print = false
   snippets     = ["${var.clc_snippets}"]
 }
 
 # Worker Container Linux configs
-data "template_file" "worker-configs" {
+data "template_file" "worker-config" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars = {
-    kubeconfig            = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    kubeconfig             = "${indent(10, var.kubeconfig)}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

bare-metal/container-linux/kubernetes/README.md

@@ -11,9 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name                    = "${var.cluster_name}"
   api_servers                     = ["${var.k8s_domain_name}"]
@@ -12,4 +12,5 @@ module "bootkube" {
   pod_cidr                        = "${var.pod_cidr}"
   service_cidr                    = "${var.service_cidr}"
   cluster_domain_suffix           = "${var.cluster_domain_suffix}"
+  enable_reporting                = "${var.enable_reporting}"
 }

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@@ -70,6 +70,10 @@ systemd:
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -86,7 +90,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -97,6 +101,7 @@ systemd:
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -123,7 +128,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/hostname
       filesystem: root
       mode: 0644
@@ -149,17 +154,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$BOOTKUBE_ASSETS \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $$RKT_OPTS \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -45,6 +45,10 @@ systemd:
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
           --mount volume=var-log,target=/var/log \
+          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
+          --mount volume=iscsiconf,target=/etc/iscsi/ \
+          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=iscsiadm,target=/sbin/iscsiadm \
           --insecure-options=image"
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
@@ -59,7 +63,7 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
@@ -69,6 +73,7 @@ systemd:
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -84,7 +89,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/outputs.tf

@@ -1,3 +1,3 @@
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
 }

bare-metal/container-linux/kubernetes/profiles.tf

@@ -160,12 +160,12 @@ data "template_file" "controller-configs" {
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.controller_domains, count.index)}"
-    etcd_name             = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.controller_domains, count.index)}"
+    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 
@@ -191,10 +191,10 @@ data "template_file" "worker-configs" {
   template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.worker_domains, count.index)}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 

bare-metal/container-linux/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -94,7 +94,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

bare-metal/container-linux/kubernetes/variables.tf

@@ -141,3 +141,9 @@ variable "kernel_args" {
   type        = "list"
   default     = []
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

bare-metal/fedora-atomic/kubernetes/README.md

@@ -11,8 +11,8 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${var.k8s_domain_name}"]
@@ -11,6 +11,7 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 
   # Fedora
   trusted_certs_dir = "/etc/pki/tls/certs"

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -40,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -51,6 +51,7 @@ write_files:
         --node-labels=node-role.kubernetes.io/master \
         --node-labels=node-role.kubernetes.io/controller="true" \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
@@ -83,9 +84,9 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -19,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -29,6 +29,7 @@ write_files:
         --network-plugin=cni \
         --node-labels=node-role.kubernetes.io/node \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
     content: |
@@ -59,7 +60,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

bare-metal/fedora-atomic/kubernetes/outputs.tf

@@ -1,3 +1,3 @@
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig}"
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
 }

bare-metal/fedora-atomic/kubernetes/profiles.tf

@@ -55,12 +55,12 @@ data "template_file" "controller-configs" {
   template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.controller_domains, count.index)}"
-    etcd_name             = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.controller_domains, count.index)}"
+    etcd_name              = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }
 
@@ -79,9 +79,9 @@ data "template_file" "worker-configs" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars {
-    domain_name           = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    domain_name            = "${element(var.worker_domains, count.index)}"
+    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
   }
 }

bare-metal/fedora-atomic/kubernetes/ssh.tf

@@ -18,7 +18,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -92,7 +92,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

bare-metal/fedora-atomic/kubernetes/variables.tf

@@ -110,3 +110,9 @@ variable "kernel_args" {
   type        = "list"
   default     = []
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

digital-ocean/container-linux/kubernetes/README.md

@@ -11,10 +11,11 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, and other [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,4 +11,5 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 }

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.10"
+            Environment="ETCD_IMAGE_TAG=v3.3.11"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -56,12 +56,9 @@ systemd:
       contents: |
         [Unit]
         Description=Kubelet via Hyperkube
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
         EnvironmentFile=/etc/kubernetes/kubelet.env
-        EnvironmentFile=/run/metadata/coreos
         Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
           --volume=resolv,kind=host,source=/etc/resolv.conf \
           --mount volume=resolv,target=/etc/resolv.conf \
@@ -89,17 +86,17 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/etc/kubernetes/kubeconfig \
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/master \
           --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
@@ -128,7 +125,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -148,17 +145,14 @@ storage:
           set -e
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
-          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \
-            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --volume assets,kind=host,source=/opt/bootkube/assets \
             --mount volume=assets,target=/assets \
             --volume bootstrap,kind=host,source=/etc/kubernetes \
             --mount volume=bootstrap,target=/etc/kubernetes \
             $${RKT_OPTS} \
-            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            quay.io/coreos/bootkube:v0.14.0 \
             --net=host \
             --dns=host \
             --exec=/bootkube -- start --asset-dir=/assets "$@"

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -31,12 +31,9 @@ systemd:
       contents: |
         [Unit]
         Description=Kubelet via Hyperkube
-        Requires=coreos-metadata.service
-        After=coreos-metadata.service
         Wants=rpc-statd.service
         [Service]
         EnvironmentFile=/etc/kubernetes/kubelet.env
-        EnvironmentFile=/run/metadata/coreos
         Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
           --volume=resolv,kind=host,source=/etc/resolv.conf \
           --mount volume=resolv,target=/etc/resolv.conf \
@@ -62,16 +59,16 @@ systemd:
           --authentication-token-webhook \
           --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_dns=${cluster_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
-          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
           --kubeconfig=/etc/kubernetes/kubeconfig \
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/node \
           --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
         ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
         Restart=always
@@ -98,7 +95,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.12.1
+          KUBELET_IMAGE_TAG=v1.13.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -116,7 +113,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.12.1 \
+            docker://k8s.gcr.io/hyperkube:v1.13.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/container-linux/kubernetes/controllers.tf

@@ -44,12 +44,18 @@ resource "digitalocean_droplet" "controllers" {
   ipv6               = true
   private_networking = true
 
-  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
+  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
   ssh_keys  = ["${var.ssh_fingerprints}"]
 
   tags = [
     "${digitalocean_tag.controllers.id}",
   ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }
 
 # Tag to label controllers
@@ -57,8 +63,16 @@ resource "digitalocean_tag" "controllers" {
   name = "${var.cluster_name}-controller"
 }
 
-# Controller Container Linux Config
-data "template_file" "controller_config" {
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
   count = "${var.controller_count}"
 
   template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
@@ -69,9 +83,9 @@ data "template_file" "controller_config" {
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
 
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster  = "${join(",", data.template_file.etcds.*.rendered)}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    etcd_initial_cluster   = "${join(",", data.template_file.etcds.*.rendered)}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 
@@ -85,11 +99,3 @@ data "template_file" "etcds" {
     dns_zone     = "${var.dns_zone}"
   }
 }
-
-data "ct_config" "controller_ign" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
-  pretty_print = false
-
-  snippets = ["${var.controller_clc_snippets}"]
-}

digital-ocean/container-linux/kubernetes/outputs.tf

@@ -1,9 +1,14 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 output "controllers_dns" {
   value = "${digitalocean_record.controllers.0.fqdn}"
 }
 
 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }
 
 output "controllers_ipv4" {

digital-ocean/container-linux/kubernetes/ssh.tf

@@ -10,7 +10,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -78,7 +78,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/container-linux/kubernetes/variables.tf

@@ -92,3 +92,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

digital-ocean/container-linux/kubernetes/workers.tf

@@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
   count = "${var.worker_count}"
 
   # DNS zone where record should be created
@@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
   value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }
 
+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
   count = "${var.worker_count}"
@@ -25,12 +37,16 @@ resource "digitalocean_droplet" "workers" {
   ipv6               = true
   private_networking = true
 
-  user_data = "${data.ct_config.worker_ign.rendered}"
+  user_data = "${data.ct_config.worker-ignition.rendered}"
   ssh_keys  = ["${var.ssh_fingerprints}"]
 
   tags = [
     "${digitalocean_tag.workers.id}",
   ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }
 
 # Tag to label workers
@@ -38,18 +54,19 @@ resource "digitalocean_tag" "workers" {
   name = "${var.cluster_name}-worker"
 }
 
-# Worker Container Linux Config
-data "template_file" "worker_config" {
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
-
-  vars = {
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  }
-}
-
-data "ct_config" "worker_ign" {
-  content      = "${data.template_file.worker_config.rendered}"
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = "${data.template_file.worker-config.rendered}"
   pretty_print = false
   snippets     = ["${var.worker_clc_snippets}"]
 }
+
+# Worker Container Linux config
+data "template_file" "worker-config" {
+  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+
+  vars = {
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+  }
+}

digital-ocean/fedora-atomic/kubernetes/README.md

@@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.12.1 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Kubernetes v1.13.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs

digital-ocean/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=81f19507faabf411db9c760d55f3d03f7d78f4c9"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c12a11c8006606b59335ecc994abe22358aaf68b"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
@@ -11,6 +11,7 @@ module "bootkube" {
   pod_cidr              = "${var.pod_cidr}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  enable_reporting      = "${var.enable_reporting}"
 
   # Fedora
   trusted_certs_dir = "/etc/pki/tls/certs"

digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -19,24 +19,9 @@ write_files:
       ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
       ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
       ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -55,7 +40,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -65,6 +50,7 @@ write_files:
         --node-labels=node-role.kubernetes.io/master \
         --node-labels=node-role.kubernetes.io/controller="true" \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
@@ -89,11 +75,10 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.10"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.11"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
   - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, enable, cloud-metadata.service]
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -1,23 +1,8 @@
 #cloud-config
 write_files:
-  - path: /etc/systemd/system/cloud-metadata.service
-    content: |
-      [Unit]
-      Description=Cloud metadata agent
-      [Service]
-      Type=oneshot
-      Environment=OUTPUT=/run/metadata/cloud
-      ExecStart=/usr/bin/mkdir -p /run/metadata
-      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
-        --url http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address\
-        --retry 10)" > $${OUTPUT}'
-      [Install]
-      WantedBy=multi-user.target
   - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
     content: |
       [Unit]
-      Requires=cloud-metadata.service
-      After=cloud-metadata.service
       Wants=rpc-statd.service
       [Service]
       ExecStartPre=/bin/mkdir -p /opt/cni/bin
@@ -34,7 +19,7 @@ write_files:
         --authentication-token-webhook \
         --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_dns=${cluster_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
         --cni-conf-dir=/etc/kubernetes/cni/net.d \
         --exit-on-lock-contention \
@@ -43,6 +28,7 @@ write_files:
         --network-plugin=cni \
         --node-labels=node-role.kubernetes.io/node \
         --pod-manifest-path=/etc/kubernetes/manifests \
+        --read-only-port=0 \
         --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
   - path: /etc/systemd/system/kubelet.path
     content: |
@@ -65,8 +51,7 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.12.1"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.3"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/controllers.tf

@@ -50,6 +50,12 @@ resource "digitalocean_droplet" "controllers" {
   tags = [
     "${digitalocean_tag.controllers.id}",
   ]
+
+  lifecycle {
+    ignore_changes = [
+      "user_data",
+    ]
+  }
 }
 
 # Tag to label controllers
@@ -71,9 +77,9 @@ data "template_file" "controller-cloudinit" {
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
     etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }
 

digital-ocean/fedora-atomic/kubernetes/outputs.tf

@@ -1,9 +1,14 @@
+output "kubeconfig-admin" {
+  value = "${module.bootkube.kubeconfig-admin}"
+}
+
 output "controllers_dns" {
   value = "${digitalocean_record.controllers.0.fqdn}"
 }
 
 output "workers_dns" {
-  value = "${digitalocean_record.workers.0.fqdn}"
+  # Multiple A and AAAA records with the same FQDN
+  value = "${digitalocean_record.workers-record-a.0.fqdn}"
 }
 
 output "controllers_ipv4" {

digital-ocean/fedora-atomic/kubernetes/ssh.tf

@@ -10,7 +10,7 @@ resource "null_resource" "copy-controller-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 
@@ -76,7 +76,7 @@ resource "null_resource" "copy-worker-secrets" {
   }
 
   provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
+    content     = "${module.bootkube.kubeconfig-kubelet}"
     destination = "$HOME/kubeconfig"
   }
 

digital-ocean/fedora-atomic/kubernetes/variables.tf

@@ -85,3 +85,9 @@ variable "cluster_domain_suffix" {
   type        = "string"
   default     = "cluster.local"
 }
+
+variable "enable_reporting" {
+  type        = "string"
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = "false"
+}

digital-ocean/fedora-atomic/kubernetes/workers.tf

@@ -1,5 +1,5 @@
 # Worker DNS records
-resource "digitalocean_record" "workers" {
+resource "digitalocean_record" "workers-record-a" {
   count = "${var.worker_count}"
 
   # DNS zone where record should be created
@@ -11,6 +11,18 @@ resource "digitalocean_record" "workers" {
   value = "${element(digitalocean_droplet.workers.*.ipv4_address, count.index)}"
 }
 
+resource "digitalocean_record" "workers-record-aaaa" {
+  count = "${var.worker_count}"
+
+  # DNS zone where record should be created
+  domain = "${var.dns_zone}"
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = "${element(digitalocean_droplet.workers.*.ipv6_address, count.index)}"
+}
+
 # Worker droplet instances
 resource "digitalocean_droplet" "workers" {
   count = "${var.worker_count}"
@@ -31,6 +43,10 @@ resource "digitalocean_droplet" "workers" {
   tags = [
     "${digitalocean_tag.workers.id}",
   ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
 }
 
 # Tag to label workers
@@ -43,8 +59,8 @@ data "template_file" "worker-cloudinit" {
   template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
 
   vars = {
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key     = "${var.ssh_authorized_key}"
+    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
   }
 }

docs/addons/heapster.md

@@ -1,6 +1,6 @@
 # Heapster
 
-[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command and Kubernetes dashboard graphs.
+[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command.
 
 ## Create
 

docs/addons/ingress.md

@@ -4,7 +4,7 @@ Nginx Ingress controller pods accept and demultiplex HTTP, HTTPS, TCP, or UDP tr
 
 ## AWS
 
-On AWS, a network load balancer (NLB) distributes traffic across a target group of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.
+On AWS, a network load balancer (NLB) distributes TCP traffic across two target groups (port 80 and 443) of worker nodes running an Ingress controller deployment. Security groups rules allow traffic to ports 80 and 443. Health checks ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -37,7 +37,7 @@ resource "google_dns_record_set" "some-application" {
 
 ## Azure
 
-On Azure, a load balancer distributes traffic across a backend pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
+On Azure, a load balancer distributes traffic across a backend address pool of worker nodes running an Ingress controller deployment. Security group rules allow traffic to ports 80 and 443. Health probes ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -101,7 +101,7 @@ resource "google_dns_record_set" "some-application" {
 
 ## Digital Ocean
 
-On Digital Ocean, a DNS A record (e.g. `nemo-workers.example.com`) resolves to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
+On Digital Ocean, DNS A and AAAA records (e.g. FQDN `nemo-workers.example.com`) resolve to each worker[^1] running an Ingress controller DaemonSet on host ports 80 and 443. Firewall rules allow IPv4 and IPv6 traffic to ports 80 and 443.
 
 Create the Ingress controller daemonset, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -124,11 +124,14 @@ resource "google_dns_record_set" "some-application" {
 }
 ```
 
+!!! note
+    Hosting IPv6 apps is possible, but requires editing the nginx-ingress addon to use `hostNetwork: true`.
+
 [^1]: Digital Ocean does offer load balancers. We've opted not to use them to keep the Digital Ocean setup simple and cheap for developers.
 
 ## Google Cloud
 
-On Google Cloud, a TCP Proxy load balancer distributes traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.
+On Google Cloud, a TCP Proxy load balancer distributes IPv4 and IPv6 TCP traffic across a backend service of worker nodes running an Ingress controller deployment. Firewall rules allow traffic to ports 80 and 443. Health check rules ensure only workers with a healthy Ingress controller receive traffic.
 
 Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, default backend, and namespace.
 
@@ -136,7 +139,7 @@ Create the Ingress controller deployment, service, RBAC roles, RBAC bindings, de
 kubectl apply -R -f addons/nginx-ingress/google-cloud
 ```
 
-For each application, add a DNS record resolving to the load balancer's IPv4 address.
+For each application, add DNS A records resolving to the load balancer's IPv4 address and DNS AAAA records resolving to the load balancer's IPv6 address.
 
 ```
 app1.example.com -> 11.22.33.44
@@ -144,10 +147,10 @@ app2.example.com -> 11.22.33.44
 app3.example.com -> 11.22.33.44
 ```
 
-Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's output `ingress_static_ipv4`. For example, you might use Terraform to manage a Google Cloud DNS record:
+Find the IPv4 address with `gcloud compute addresses list` or use the Typhoon module's outputs `ingress_static_ipv4` and `ingress_static_ipv6`. For example, you might use Terraform to manage a Google Cloud DNS record:
 
 ```tf
-resource "google_dns_record_set" "some-application" {
+resource "google_dns_record_set" "app-record-a" {
   # DNS zone name
   managed_zone = "example-zone"
 
@@ -157,4 +160,15 @@ resource "google_dns_record_set" "some-application" {
   ttl     = 300
   rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv4}"]
 }
+
+resource "google_dns_record_set" "app-record-aaaa" {
+  # DNS zone name
+  managed_zone = "example-zone"
+
+  # DNS record
+  name    = "app.example.com."
+  type    = "AAAA"
+  ttl     = 300
+  rrdatas = ["${module.google-cloud-yavin.ingress_static_ipv6}"]
+}
 ```

docs/advanced/worker-pools.md

@@ -16,7 +16,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo
 
 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.13.3"
   
   providers = {
     aws = "aws.default"
@@ -67,7 +67,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
 | count | Number of instances | 1 | 3 |
-| instance_type | EC2 instance type | "t2.small" | "t2.medium" |
+| instance_type | EC2 instance type | "t3.small" | "t3.medium" |
 | os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | spot_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
@@ -82,7 +82,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define
 
 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.13.3"
   
   providers = {
     azurerm = "azurerm.default"
@@ -152,7 +152,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu
 
 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.12.1"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.13.3"
 
   providers = {
     google = "google.default"
@@ -187,11 +187,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.12.1
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.12.1
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.12.1
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.12.1
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.12.1
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.13.3
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.13.3
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.13.3
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.13.3
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.13.3
 ```
 
 ### Variables

docs/announce.md

@@ -18,7 +18,7 @@ Fedora Atomic is a container-optimized operating system designed for large-scale
 
 For newcomers, Typhoon is a free (cost and freedom) Kubernetes distribution providing upstream Kubernetes, declarative configuration via [Terraform](https://www.terraform.io/intro/index.html), and support for AWS, Google Cloud, DigitalOcean, and bare-metal. Typhoon clusters use a [self-hosted](https://github.com/kubernetes-incubator/bootkube) control plane, support [Calico](https://www.projectcalico.org/blog/) and [flannel](https://coreos.com/flannel/docs/latest/) CNI networking, and enable etcd TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/), and network policy.
 
-Typhoon for Fedora Atomic reflects many of the same principles that created Typhoon for Container Linux. Clusters are declared using plain Terraform configs that can be versioned. In lieu of Ignition, instances are declaratively provisioned with Cloud-Init and kickstart (bare-metal only). TLS assets are generated. Hosts run only a kubelet service, other components are scheduled (i.e. self-hosted). The upstream hyperkube is used directly[^1]. And clusters are kept minimal by offering optional addons for [Ingress](https://typhoon.psdn.io/addons/ingress/), [Prometheus](https://typhoon.psdn.io/addons/prometheus/), and [Grafana](https://typhoon.psdn.io/addons/grafana/). Typhoon compliments and enhances Fedora Atomic as a choice of operating system for Kubernetes.
+Typhoon for Fedora Atomic reflects many of the same principles that created Typhoon for Container Linux. Clusters are declared using plain Terraform configs that can be versioned. In lieu of Ignition, instances are declaratively provisioned with Cloud-Init and kickstart (bare-metal only). TLS assets are generated. Hosts run only a kubelet service, other components are scheduled (i.e. self-hosted). The upstream hyperkube is used directly[^1]. And clusters are kept minimal by offering optional addons for [Ingress](/addons/ingress/), [Prometheus](/addons/prometheus/), and [Grafana](/addons/grafana/). Typhoon compliments and enhances Fedora Atomic as a choice of operating system for Kubernetes.
 
 Meanwhile, Fedora Atomic adds some promising new low-level technologies:
 

docs/architecture/aws.md

@@ -0,0 +1,13 @@
+# AWS
+
+## IPv6
+
+Status of IPv6 on Typhoon AWS clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | No        |
+
+* AWS Network Load Balancers do not support `dualstack`.

docs/architecture/azure.md

@@ -0,0 +1,13 @@
+# Azure
+
+## IPv6
+
+Status of IPv6 on Typhoon Azure clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | No        |
+| Node Outbound IPv6      | No        |
+| Kubernetes Ingress IPv6 | No        |
+
+* Azure does not allow reserving a static IPv6 address 

docs/architecture/bare-metal.md

@@ -0,0 +1,13 @@
+# Bare-Metal
+
+## IPv6
+
+Status of IPv6 on Typhoon bare-metal clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Possible  |
+
+IPv6 support depends upon the bare-metal network environment.

docs/architecture/concepts.md

@@ -69,7 +69,7 @@ Module versioning ensures `terraform get --update` only fetches the desired vers
 
 Maintain Terraform configs for "live" infrastructure in a versioned repository. Seek to organize configs to reflect resources that should be managed together in a `terraform apply` invocation.
 
-You may choose to organize resources all together, by team, by project, or some other scheme. Here's an example that manages four clusters together:
+You may choose to organize resources all together, by team, by project, or some other scheme. Here's an example that manages clusters together:
 
 ```sh
 .git/

docs/architecture/digitalocean.md

@@ -0,0 +1,11 @@
+# DigitalOcean
+
+## IPv6
+
+Status of IPv6 on Typhoon DigitalOcean clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | Yes       |
+| Node Outbound IPv6      | Yes       |
+| Kubernetes Ingress IPv6 | Possible  |

docs/architecture/google-cloud.md

@@ -0,0 +1,11 @@
+# Google Cloud
+
+## IPv6
+
+Status of IPv6 on Typhoon Google Cloud clusters.
+
+| IPv6 Feature            | Supported |
+|-------------------------|-----------|
+| Node IPv6 address       | No        |
+| Node Outbound IPv6      | No        |
+| Kubernetes Ingress IPv6 | Yes       |