Add missing changelog entry for Digital Ocean fix

Set module version tutorials docs for good practice
Update bootkube and terraform-render-bootkube to v0.10.0
2025-08-03 04:21:34 +02:00 · 2018-01-20 07:52:40 -08:00 · 2018-01-19 23:16:48 -08:00 · 2018-01-19 23:10:45 -08:00 · 2018-01-19 08:40:09 -08:00 · 2018-01-19 08:33:26 -08:00
36 changed files with 520 additions and 122 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,9 +4,50 @@ Notable changes between versions.

 ## Latest

+## v1.9.2
+
+* Kubernetes [v1.9.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v192)
+* Add Terraform v0.11.x support
+  * Add explicit "providers" section to modules for Terraform v0.11.x
+  * Retain support for Terraform v0.10.4+
+* Add [migration guide](https://github.com/poseidon/typhoon/blob/master/docs/topics/maintenance.md) from Terraform v0.10.x to v0.11.x (**action required!**)
+* Update etcd from 3.2.13 to 3.2.14
+* Update calico from 2.6.5 to 2.6.6
+* Update kube-dns from v1.14.7 to v1.14.8
+* Use separate service account for kube-dns
+* Use kubernetes-incubator/bootkube v0.10.0
+
+#### Addons
+
+* Update CLUO to v0.5.0 to fix compatibility with Kubernetes 1.9 (**important**)
+  * Earlier versions can't roll out Container Linux updates on Kubernetes 1.9 nodes ([cluo#163](https://github.com/coreos/container-linux-update-operator/issues/163))
+* Update kube-state-metrics from v1.1.0 to v1.2.0
+* Fix RBAC cluster role for kube-state-metrics
+
+#### Bare-Metal
+
+* Use per-node Container Linux install profiles ([#97](https://github.com/poseidon/typhoon/pull/97))
+  * Allow Container Linux channel/version to be chosen per-cluster
+  * Fix issue where cluster deletion could require `terraform apply` multiple times
+
+#### Digital Ocean
+
+* Relax `digitalocean` provider version constraint
+* Fix bug with `terraform plan` always showing a firewall diff to be applied ([#3](https://github.com/poseidon/typhoon/issues/3))
+
+## v1.9.1
+
+* Kubernetes [v1.9.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v191)
+* Update kube-dns from 1.14.5 to v1.14.7
+* Update etcd from 3.2.0 to 3.2.13
+* Update Calico from v2.6.4 to v2.6.5
+* Enable portmap to fix hostPort with Calico
+* Use separate service account for controller-manager
+
 ## v1.8.6

 * Kubernetes [v1.8.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#v186)
+* Update Calico from v2.6.3 to v2.6.4

 ## v1.8.5

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.9.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
@ -45,11 +45,19 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 module "google-cloud-yavin" {
  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"
  
+  providers = {
+    google = "google.default"
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }
+
  # Google Cloud
  region        = "us-central1"
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"
-  os_image      = "coreos-stable-1576-4-0-v20171206"
+  os_image      = "coreos-stable-1576-5-0-v20180105"

  cluster_name       = "yavin"
  controller_count   = 1
@ -75,12 +83,12 @@ Apply complete! Resources: 37 added, 0 changed, 0 destroyed.
 In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.

 ```sh
-$ KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
+$ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.8.6
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.6
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.6
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.9.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.9.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.9.2
 ```

 List the pods.
--- a/addons/cluo/update-agent.yaml
+++ b/addons/cluo/update-agent.yaml
@ -15,7 +15,7 @@ spec:
    spec:
      containers:
      - name: update-agent
-        image: quay.io/coreos/container-linux-update-operator:v0.4.1
+        image: quay.io/coreos/container-linux-update-operator:v0.5.0
        command:
        - "/bin/update-agent"
        volumeMounts:
--- a/addons/cluo/update-operator.yaml
+++ b/addons/cluo/update-operator.yaml
@ -12,7 +12,7 @@ spec:
    spec:
      containers:
      - name: update-operator
-        image: quay.io/coreos/container-linux-update-operator:v0.4.1
+        image: quay.io/coreos/container-linux-update-operator:v0.5.0
        command:
        - "/bin/update-operator"
        env:
--- a/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
@ -12,7 +12,9 @@ rules:
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
+  - persistentvolumes
  - namespaces
+  - endpoints
  verbs: ["list", "watch"]
 - apiGroups: ["extensions"]
  resources:
@ -29,3 +31,8 @@ rules:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
+- apiGroups: ["autoscaling"]
+  resources:
+  - horizontalpodautoscalers
+  verbs: ["list", "watch"]
+
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -22,7 +22,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.1.0
+        image: quay.io/coreos/kube-state-metrics:v1.2.0
        ports:
          - name: metrics
            containerPort: 8080
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.9.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/aws/container-linux/kubernetes/bootkube.tf
+++ b/aws/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.10.0"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.2.0"
+            Environment="ETCD_IMAGE_TAG=v3.2.14"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -46,7 +46,7 @@ systemd:
      enable: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -129,7 +129,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -150,7 +150,7 @@ storage:
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.10.0}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/aws/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -22,7 +22,7 @@ systemd:
      enable: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -103,7 +103,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -121,7 +121,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.8.6 \
+            docker://gcr.io/google_containers/hyperkube:v1.9.2 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.9.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/bare-metal/container-linux/kubernetes/bootkube.tf
+++ b/bare-metal/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.10.0"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${var.k8s_domain_name}"]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.2.0"
+            Environment="ETCD_IMAGE_TAG=v3.2.14"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@ -54,7 +54,7 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -117,7 +117,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/hostname
      filesystem: root
      mode: 0644
@ -144,7 +144,7 @@ storage:
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.10.0}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -30,7 +30,7 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -82,7 +82,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/kubernetes/groups.tf
+++ b/bare-metal/container-linux/kubernetes/groups.tf
@ -3,7 +3,7 @@ resource "matchbox_group" "container-linux-install" {
  count = "${length(var.controller_names) + length(var.worker_names)}"

  name    = "${format("container-linux-install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
-  profile = "${var.cached_install == "true" ? matchbox_profile.cached-container-linux-install.name : matchbox_profile.container-linux-install.name}"
+  profile = "${var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"

  selector {
    mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -1,6 +1,8 @@
 // Container Linux Install profile (from release.core-os.net)
 resource "matchbox_profile" "container-linux-install" {
-  name   = "container-linux-install"
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+  name  = "${format("%s-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+
  kernel = "http://${var.container_linux_channel}.release.core-os.net/amd64-usr/${var.container_linux_version}/coreos_production_pxe.vmlinuz"

  initrd = [
@ -16,10 +18,12 @@ resource "matchbox_profile" "container-linux-install" {
    "${var.kernel_args}",
  ]

-  container_linux_config = "${data.template_file.container-linux-install-config.rendered}"
+  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
 }

-data "template_file" "container-linux-install-config" {
+data "template_file" "container-linux-install-configs" {
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+
  template = "${file("${path.module}/cl/container-linux-install.yaml.tmpl")}"

  vars {
@ -37,7 +41,9 @@ data "template_file" "container-linux-install-config" {
 // Container Linux Install profile (from matchbox /assets cache)
 // Note: Admin must have downloaded container_linux_version into matchbox assets.
 resource "matchbox_profile" "cached-container-linux-install" {
-  name   = "cached-container-linux-install"
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+  name  = "${format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+
  kernel = "/assets/coreos/${var.container_linux_version}/coreos_production_pxe.vmlinuz"

  initrd = [
@ -53,10 +59,12 @@ resource "matchbox_profile" "cached-container-linux-install" {
    "${var.kernel_args}",
  ]

-  container_linux_config = "${data.template_file.cached-container-linux-install-config.rendered}"
+  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
 }

-data "template_file" "cached-container-linux-install-config" {
+data "template_file" "cached-container-linux-install-configs" {
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+
  template = "${file("${path.module}/cl/container-linux-install.yaml.tmpl")}"

  vars {
--- a/bare-metal/container-linux/pxe-worker/cl/bootkube-worker.yaml.tmpl
+++ b/bare-metal/container-linux/pxe-worker/cl/bootkube-worker.yaml.tmpl
@ -30,7 +30,7 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -98,7 +98,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.9.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/digital-ocean/container-linux/kubernetes/bootkube.tf
+++ b/digital-ocean/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.10.0"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.2.0"
+            Environment="ETCD_IMAGE_TAG=v3.2.14"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -54,7 +54,7 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Requires=coreos-metadata.service
        After=coreos-metadata.service
        Wants=rpc-statd.service
@ -120,7 +120,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -141,7 +141,7 @@ storage:
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.10.0}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -30,7 +30,7 @@ systemd:
    - name: kubelet.service
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Requires=coreos-metadata.service
        After=coreos-metadata.service
        Wants=rpc-statd.service
@ -94,7 +94,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -112,7 +112,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.8.6 \
+            docker://gcr.io/google_containers/hyperkube:v1.9.2 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/network.tf
+++ b/digital-ocean/container-linux/kubernetes/network.tf
@ -22,12 +22,12 @@ resource "digitalocean_firewall" "rules" {
    },
    {
      protocol    = "udp"
-      port_range  = "all"
+      port_range  = "1-65535"
      source_tags = ["${digitalocean_tag.controllers.name}", "${digitalocean_tag.workers.name}"]
    },
    {
      protocol    = "tcp"
-      port_range  = "all"
+      port_range  = "1-65535"
      source_tags = ["${digitalocean_tag.controllers.name}", "${digitalocean_tag.workers.name}"]
    },
  ]
@ -35,17 +35,18 @@ resource "digitalocean_firewall" "rules" {
  # allow all outbound traffic
  outbound_rule = [
    {
-      protocol              = "icmp"
+      protocol              = "tcp"
+      port_range            = "1-65535"
      destination_addresses = ["0.0.0.0/0", "::/0"]
    },
    {
      protocol              = "udp"
-      port_range            = "all"
+      port_range            = "1-65535"
      destination_addresses = ["0.0.0.0/0", "::/0"]
    },
    {
-      protocol              = "tcp"
-      port_range            = "all"
+      protocol              = "icmp"
+      port_range            = "1-65535"
      destination_addresses = ["0.0.0.0/0", "::/0"]
    },
  ]
--- a/digital-ocean/container-linux/kubernetes/require.tf
+++ b/digital-ocean/container-linux/kubernetes/require.tf
@ -5,7 +5,7 @@ terraform {
 }

 provider "digitalocean" {
-  version = "0.1.2"
+  version = "~> 0.1.2"
 }

 provider "local" {
--- a/docs/addons/cluo.md
+++ b/docs/addons/cluo.md
@ -18,7 +18,7 @@ kubectl apply -f addons/cluo -R
 $ kubectl get nodes --show-labels
 ...
 container-linux-update.v1.coreos.com/group=stable
-container-linux-update.v1.coreos.com/version=1576.4.0
+container-linux-update.v1.coreos.com/version=1576.5.0
 ```

 `update-operator` ensures one node reboots at a time and that pods are drained prior to reboot.
--- a/docs/aws.md
+++ b/docs/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.8.6 cluster on AWS.
+In this tutorial, we'll create a Kubernetes v1.9.2 cluster on AWS.

 We'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module. On apply, a VPC, gateway, subnets, auto-scaling groups of controllers and workers, network load balancers for controllers and workers, and security groups will be created.

@ -10,15 +10,15 @@ Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube

 * AWS Account and IAM credentials
 * AWS Route53 DNS Zone (registered Domain Name or delegated subdomain)
-* Terraform v0.10.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
+* Terraform v0.11.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your system.

 ```sh
 $ terraform version
-Terraform v0.10.7
+Terraform v0.11.1
 ```

 Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
@ -57,9 +57,32 @@ Configure the AWS provider to use your access key credentials in a `providers.tf

 ```tf
 provider "aws" {
+  version = "~> 1.5.0"
+  alias   = "default"
+
  region                  = "eu-central-1"
  shared_credentials_file = "/home/user/.config/aws/credentials"
 }
+
+provider "local" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "null" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "template" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+  alias = "default"
+}
 ```

 Additional configuration options are described in the `aws` provider [docs](https://www.terraform.io/docs/providers/aws/).
@ -73,7 +96,15 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.9.2"
+
+  providers = {
+    aws = "aws.default"
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }
  
  cluster_name = "tempest"

@ -119,7 +150,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.10.0 (update)
 ```

 Plan the resources to be created.
@ -148,12 +179,12 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 [Install kubectl](https://coreos.com/kubernetes/docs/latest/configure-kubectl.html) on your system. Use the generated `kubeconfig` credentials to access the Kubernetes cluster and list nodes.

 ```
-$ KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
+$ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.8.6
-ip-10-0-19-112   Ready     34m       v1.8.6
-ip-10-0-4-22     Ready     34m       v1.8.6
+ip-10-0-12-221   Ready     34m       v1.9.2
+ip-10-0-19-112   Ready     34m       v1.9.2
+ip-10-0-4-22     Ready     34m       v1.9.2
 ```

 List the pods.
@ -179,7 +210,7 @@ kube-system   pod-checkpointer-4kxtl-ip-10-0-12-221     1/1    Running   0

 ## Going Further

-Learn about [version pinning](concepts.md#versioning), maintenance, and [addons](addons/overview.md).
+Learn about [version pinning](concepts.md#versioning), [maintenance](topics/maintenance.md), and [addons](addons/overview.md).

 !!! note
    On Container Linux clusters, install the `container-linux-update-operator` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
--- a/docs/bare-metal.md
+++ b/docs/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.8.6 cluster on bare-metal.
+In this tutorial, we'll network boot and provision a Kubernetes v1.9.2 cluster on bare-metal.

 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers.

@ -12,7 +12,7 @@ Controllers are provisioned as etcd peers and run `etcd-member` (etcd3) and `kub
 * PXE-enabled [network boot](https://coreos.com/matchbox/docs/latest/network-setup.html) environment
 * Matchbox v0.6+ deployment with API enabled
 * Matchbox credentials `client.crt`, `client.key`, `ca.crt`
-* Terraform v0.10.x and [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) installed locally
+* Terraform v0.11.x and [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) installed locally

 ## Machines

@ -109,11 +109,11 @@ Read about the [many ways](https://coreos.com/matchbox/docs/latest/network-setup

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your system.

 ```sh
 $ terraform version
-Terraform v0.10.7
+Terraform v0.11.1
 ```

 Add the [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin binary for your system.
@ -149,6 +149,26 @@ provider "matchbox" {
  client_key  = "${file("~/.config/matchbox/client.key")}"
  ca          = "${file("~/.config/matchbox/ca.crt")}"
 }
+
+provider "local" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "null" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "template" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+  alias = "default"
+}
 ```

 ## Cluster
@ -157,12 +177,19 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.9.2"
+  
+  providers = {
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }
  
  # install
  matchbox_http_endpoint  = "http://matchbox.example.com"
  container_linux_channel = "stable"
-  container_linux_version = "1576.4.0"
+  container_linux_version = "1576.5.0"
  ssh_authorized_key      = "ssh-rsa AAAAB3Nz..."

  # cluster
@ -219,7 +246,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.10.0 (update)
 ```

 Plan the resources to be created.
@ -287,12 +314,12 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 [Install kubectl](https://coreos.com/kubernetes/docs/latest/configure-kubectl.html) on your system. Use the generated `kubeconfig` credentials to access the Kubernetes cluster and list nodes.

 ```
-$ KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
+$ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.8.6
-node2.example.com   Ready     11m       v1.8.6
-node3.example.com   Ready     11m       v1.8.6
+node1.example.com   Ready     11m       v1.9.2
+node2.example.com   Ready     11m       v1.9.2
+node3.example.com   Ready     11m       v1.9.2
 ```

 List the pods.
@ -319,7 +346,7 @@ kube-system   pod-checkpointer-wf65d-node1.example.com   1/1       Running   0

 ## Going Further

-Learn about [version pinning](concepts.md#versioning), maintenance, and [addons](addons/overview.md).
+Learn about [version pinning](concepts.md#versioning), [maintenance](topics/maintenance.md), and [addons](addons/overview.md).

 !!! note
    On Container Linux clusters, install the `container-linux-update-operator` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@ -332,7 +359,7 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]
 |:-----|:------------|:--------|
 | matchbox_http_endpoint | Matchbox HTTP read-only endpoint | http://matchbox.example.com:8080 |
 | container_linux_channel | Container Linux channel | stable, beta, alpha |
-| container_linux_version | Container Linux version of the kernel/initrd to PXE and the image to install | 1576.4.0 |
+| container_linux_version | Container Linux version of the kernel/initrd to PXE and the image to install | 1576.5.0 |
 | cluster_name | Cluster name | mercury |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
 | ssh_authorized_key | SSH public key for ~/.ssh/authorized_keys | "ssh-rsa AAAAB3Nz..." |
--- a/docs/digital-ocean.md
+++ b/docs/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.8.6 cluster on Digital Ocean.
+In this tutorial, we'll create a Kubernetes v1.9.2 cluster on Digital Ocean.

 We'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module. On apply, firewall rules, DNS records, tags, and droplets for Kubernetes controllers and workers will be created.

@ -10,15 +10,15 @@ Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube

 * Digital Ocean Account and Token
 * Digital Ocean Domain (registered Domain Name or delegated subdomain)
-* Terraform v0.10.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
+* Terraform v0.11.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your system.

 ```sh
 $ terraform version
-Terraform v0.10.7
+Terraform v0.11.1
 ```

 Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
@ -58,7 +58,29 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.

 ```tf
 provider "digitalocean" {
+  version = "0.1.3"
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
+  alias = "default"
+}
+
+provider "local" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "null" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "template" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+  alias = "default"
 }
 ```

@ -68,7 +90,15 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube

 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.9.2"
+  
+  providers = {
+    digitalocean = "digitalocean.default"
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }

  region   = "nyc3"
  dns_zone = "digital-ocean.example.com"
@ -114,7 +144,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.10.0 (update)
 ```

 Plan the resources to be created.
@ -144,12 +174,12 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 [Install kubectl](https://coreos.com/kubernetes/docs/latest/configure-kubectl.html) on your system. Use the generated `kubeconfig` credentials to access the Kubernetes cluster and list nodes.

 ```
-$ KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
+$ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.8.6
-10.132.115.81    Ready     10m       v1.8.6
-10.132.124.107   Ready     10m       v1.8.6
+10.132.110.130   Ready     10m       v1.9.2
+10.132.115.81    Ready     10m       v1.9.2
+10.132.124.107   Ready     10m       v1.9.2
 ```

 List the pods.
@ -174,7 +204,7 @@ kube-system   pod-checkpointer-pr1lq-10.132.115.81       1/1       Running   0

 ## Going Further

-Learn about [version pinning](concepts.md#versioning), maintenance, and [addons](addons/overview.md).
+Learn about [version pinning](concepts.md#versioning), [maintenance](topics/maintenance.md), and [addons](addons/overview.md).

 !!! note
    On Container Linux clusters, install the `container-linux-update-operator` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
--- a/docs/google-cloud.md
+++ b/docs/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.8.6 cluster on Google Compute Engine (not GKE).
+In this tutorial, we'll create a Kubernetes v1.9.2 cluster on Google Compute Engine (not GKE).

 We'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module. On apply, a network, firewall rules, managed instance groups of Kubernetes controllers and workers, network load balancers for controllers and workers, and health checks will be created.

@ -10,15 +10,15 @@ Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube

 * Google Cloud Account and Service Account
 * Google Cloud DNS Zone (registered Domain Name or delegated subdomain)
-* Terraform v0.10.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
+* Terraform v0.11.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.11.x on your system.

 ```sh
 $ terraform version
-Terraform v0.10.7
+Terraform v0.11.1
 ```

 Add the [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) plugin binary for your system.
@ -57,10 +57,33 @@ Configure the Google Cloud provider to use your service account key, project-id,

 ```tf
 provider "google" {
+  version = "1.2"
+  alias   = "default"
+
  credentials = "${file("~/.config/google-cloud/terraform.json")}"
  project     = "project-id"
  region      = "us-central1"
 }
+
+provider "local" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "null" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "template" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+  alias = "default"
+}
 ```

 Additional configuration options are described in the `google` provider [docs](https://www.terraform.io/docs/providers/google/index.html).
@ -74,13 +97,21 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.9.2"
+  
+  providers = {
+    google = "google.default"
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }

  # Google Cloud
  region        = "us-central1"
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"
-  os_image      = "coreos-stable-1576-4-0-v20171206"
+  os_image      = "coreos-stable-1576-5-0-v20180105"

  cluster_name       = "yavin"
  controller_count   = 1
@ -120,7 +151,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.10.0 (update)
 ```

 Plan the resources to be created.
@ -151,12 +182,12 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 [Install kubectl](https://coreos.com/kubernetes/docs/latest/configure-kubectl.html) on your system. Use the generated `kubeconfig` credentials to access the Kubernetes cluster and list nodes.

 ```
-$ KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
+$ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.8.6
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.6
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.6
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.9.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.9.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.9.2
 ```

 List the pods.
@ -181,7 +212,7 @@ kube-system   pod-checkpointer-l6lrt                    1/1    Running   0

 ## Going Further

-Learn about [version pinning](concepts.md#versioning), maintenance, and [addons](addons/overview.md).
+Learn about [version pinning](concepts.md#versioning), [maintenance](topics/maintenance.md), and [addons](addons/overview.md).

 !!! note
    On Container Linux clusters, install the `container-linux-update-operator` addon to coordinate reboots and drains when nodes auto-update. Otherwise, updates may not be applied until the next reboot.
@ -197,7 +228,7 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
 | ssh_authorized_key | SSH public key for ~/.ssh_authorized_keys | "ssh-rsa AAAAB3NZ..." |
-| os_image | OS image for compute instances | "coreos-stable-1576-4-0-v20171206" |
+| os_image | OS image for compute instances | "coreos-stable-1576-5-0-v20180105" |
 | asset_dir | Path to a directory where generated assets should be placed (contains secrets) | "/home/user/.secrets/clusters/yavin" |

 Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Container Linux [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep coreos`.
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.9.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics and other optional [addons](addons/overview.md)
@ -45,11 +45,19 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 module "google-cloud-yavin" {
  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"
  
+  providers = {
+    google = "google.default"
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }
+
  # Google Cloud
  region        = "us-central1"
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"
-  os_image      = "coreos-stable-1576-4-0-v20171206"
+  os_image      = "coreos-stable-1576-5-0-v20180105"

  cluster_name       = "yavin"
  controller_count   = 1
@ -74,12 +82,12 @@ Apply complete! Resources: 64 added, 0 changed, 0 destroyed.
 In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.

 ```
-$ KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
+$ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.8.6
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.6
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.6
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.9.2
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.9.2
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.9.2
 ```

 List the pods.
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -0,0 +1,204 @@
+# Maintenance
+
+## Best Practices
+
+* Run multiple Kubernetes clusters. Run across platforms. Plan for regional and cloud outages.
+* Require applications be platform agnostic. Moving an application between a Kubernetes AWS cluster and a Kubernetes bare-metal cluster should be normal.
+* Strive to make single-cluster outages tolerable. Practice performing failovers.
+* Strive to make single-cluster outages a non-event. Load balance applications between multiple clusters, automate failover behaviors, and adjust alerting behaviors.
+
+## Versioning
+
+Typhoon provides tagged releases to allow clusters to be versioned using ordinary Terraform configs.
+
+```
+module "google-cloud-yavin" {
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.8.6"
+  ...
+}
+
+module "bare-metal-mercury" {
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.9.2"
+  ...
+}
+```
+
+Master is updated regularly, so it is recommended to [pin](https://www.terraform.io/docs/modules/sources.html) modules to a [release tag](https://github.com/poseidon/typhoon/releases) or [commit](https://github.com/poseidon/typhoon/commits/master) hash. Pinning ensures `terraform get --update` only fetches the desired version.
+
+## Upgrades
+
+Typhoon recommends upgrading clusters using a blue-green replacement strategy and migrating workloads.
+
+1. Launch new (candidate) clusters from tagged releases
+2. Apply workloads from existing cluster(s)
+3. Evaluate application health and performance
+4. Migrate application traffic to the new cluster
+5. Compare metrics and delete old cluster when ready
+
+Blue-green replacement reduces risk for clusters running critical applications. Candidate clusters allow baseline properties of clusters to be assessed (e.g. pod-to-pod bandwidth). Applying application workloads allows health to be assessed before being subjected to traffic (e.g. detect any changes in Kubernetes behavior between versions). Migration to the new cluster can be controlled according to requirements. Migration may mean updating DNS records to resolve the new cluster's ingress or may involve a load balancer gradually shifting traffic to the new cluster "backend". Retain the old cluster for a time to compare metrics or for fallback if issues arise.
+
+Blue-green replacement provides some subtler benefits as well:
+
+* Encourages investment in tooling for traffic migration and failovers. When a cluster incident arises, shifting applications to a healthy cluster will be second nature.
+* Discourages reliance on in-place opqaue state. Retain confidence in your ability to create infrastructure from scratch.
+* Allows Typhoon to make architecture changes between releases and eases the burden on Typhoon maintainers. By contrast, distros promising in-place upgrades get stuck with their mistakes or require complex and error-prone migrations.
+
+### Bare-Metal
+
+Typhoon bare-metal clusters are provisioned by a PXE-enabled network boot environment and a [Matchbox](https://github.com/coreos/matchbox) service. To upgrade, re-provision machines into a new cluster.
+
+Failover application workloads to another cluster (varies).
+
+```
+kubectl config use-context other-context
+kubectl apply -f mercury -R
+# DNS or load balancer changes
+```
+
+Power off bare-metal machines and set their next boot device to PXE.
+
+```
+ipmitool -H node1.example.com -U USER -P PASS power off
+ipmitool -H node1.example.com -U USER -P PASS chassis bootdev pxe
+```
+
+Delete or comment the Terraform config for the cluster.
+
+```
+- module "bare-metal-mercury" {
+-   source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes"
+-   ...
+-}
+```
+
+Apply to delete old provisioning configs from Matchbox.
+
+```
+$ terraform apply  
+Apply complete! Resources: 0 added, 0 changed, 55 destroyed.
+```
+
+Re-provision a new cluster by following the bare-metal [tutorial](../bare-metal.md#cluster).
+
+### Cloud
+
+Create a new cluster following the tutorials. Failover application workloads to the new cluster (varies).
+
+```
+kubectl config use-context other-context
+kubectl apply -f mercury -R
+# DNS or load balancer changes
+```
+
+Once you're confident in the new cluster, delete the Terraform config for the old cluster.
+
+```
+- module "google-cloud-yavin" {
+-   source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"
+-   ...
+-}
+```
+
+Apply to delete the cluster.
+
+```
+$ terraform apply  
+Apply complete! Resources: 0 added, 0 changed, 55 destroyed.
+```
+
+### Alternatives
+
+#### In-place Edits
+
+Typhoon uses a self-hosted Kubernetes control plane which allows certain manifest upgrades to be performed in-place. Components like `apiserver`, `controller-manager`, `scheduler`, `flannel`/`calico`, `kube-dns`, and `kube-proxy` are run on Kubernetes itself and can be edited via `kubectl`. If you're interested, see the bootkube [upgrade docs](https://github.com/kubernetes-incubator/bootkube/blob/master/Documentation/upgrading.md).
+
+In certain scenarios, in-place edits can be useful for quickly rolling out security patches (e.g. bumping `kube-dns`) or prioritizing speed over the safety of a proper cluster re-provision and transition.
+
+!!! note
+    Rarely, we may test certain security in-place edits and mention them as an option in release notes.
+
+!!! warning
+    Typhoon does not support or document in-place edits as an upgrade strategy. They involve inherent risks and we choose not to make recommendations or guarentees about the safety of different in-place upgrades. Its explicitly a non-goal.
+
+#### Node Replacement
+
+Typhoon supports multi-controller clusters, so it is possible to upgrade a cluster by deleting and replacing nodes one by one.
+
+!!! warning
+    Typhoon does not support or document node replacement as an upgrade strategy. It limits Typhoon's ability to make infrastructure and architectural changes between tagged releases. 
+
+## Terraform v0.11.x
+
+Terraform v0.10.x to v0.11.x introduced breaking changes in the provider and module inheritance relationship that you MUST be aware of when upgrading to the v0.11.x `terraform` binary. Terraform now allows multiple named (i.e. aliased) copies of a provider to exist (e.g `aws.default`, `aws.somename`). Terraform now also requires providers be explicitly passed to modules in order to satisfy module version contraints (which Typhoon modules define). Full details can be found in [typhoon#77](https://github.com/poseidon/typhoon/issues/77) and [hashicorp#16824](https://github.com/hashicorp/terraform/issues/16824).
+
+In particular, after upgrading to the v0.11.x `terraform` binary, you'll notice:
+
+* `terraform plan` does not succeed and prompts for variables when it didn't before
+* `terraform plan` does not succeed and mentions "provider configuration block is required for all operations"
+* `terraform apply` fails when you comment or remove a module usage in order to delete a cluster
+
+### New users
+
+New users can start with Terraform v0.11.x and follow the Typhoon docs without issue.
+
+### Existing
+
+Users who used modules to create clusters with Terraform v0.10.x and still manage those clusters via Terraform must explicitly add each provider used in `provider.tf`:
+
+```
+provider "local" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "null" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "template" {
+  version = "~> 1.0"
+  alias = "default"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+  alias = "default"
+}
+```
+
+Modify the `google`, `aws`, or `digitalocean` provider section to specify an explicit `alias` name.
+
+```
+provider "digitalocean" {
+  version = "0.1.2"
+  token = "${chomp(file("~/.config/digital-ocean/token"))}"
+  alias = "default"
+}
+```
+
+!!! note
+    In these examples, we've chosen to name each provider "default", though the point of the Terraform changes is that other possibilities are possible.
+
+Edit each instance (i.e. usage) of a module and explicitly pass the providers.
+
+```
+module "aws-cluster" {
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes"
+  
+  providers = {
+    aws = "aws.default"
+    local = "local.default"
+    null = "null.default"
+    template = "template.default"
+    tls = "tls.default"
+  }
+
+  cluster_name = "somename"
+```
+
+Re-run `terraform plan`. Plan will claim there are no changes to apply. Run `terraform apply` anyway as this will update Terraform state to be aware of the explicit provider versions.
+
+### Verify
+
+You should now be able to run `terraform plan` without errors. When you choose, you may comment or delete a module from Terraform configs and `terraform apply` should destroy the cluster correctly.
--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.9.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/google-cloud/container-linux/kubernetes/bootkube.tf
+++ b/google-cloud/container-linux/kubernetes/bootkube.tf
@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.10.0"

  cluster_name          = "${var.cluster_name}"
  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
--- a/google-cloud/container-linux/kubernetes/controllers/cl/controller.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/controllers/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.2.0"
+            Environment="ETCD_IMAGE_TAG=v3.2.14"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -46,7 +46,7 @@ systemd:
      enable: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -130,7 +130,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -151,7 +151,7 @@ storage:
          # Move experimental manifests
          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.10.0}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -22,7 +22,7 @@ systemd:
      enable: true
      contents: |
        [Unit]
-        Description=Kubelet via Hyperkube ACI
+        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
        EnvironmentFile=/etc/kubernetes/kubelet.env
@ -104,7 +104,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.6
+          KUBELET_IMAGE_TAG=v1.9.2
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -122,7 +122,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.8.6 \
+            docker://gcr.io/google_containers/hyperkube:v1.9.2 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/mkdocs.yml
+++ b/mkdocs.yml
@ -1,13 +1,11 @@
 site_name: Typhoon
-theme: material
-site_favicon: 'img/favicon.ico'
-repo_name: 'poseidon/typhoon'
-repo_url: 'https://github.com/poseidon/typhoon'
-extra:
+theme:
+  name: 'material'
  palette:
    primary: 'blue'
    accent: 'light blue'
  logo: 'img/spin.png'
+  favicon: 'img/favicon.ico'
  font:
    text: 'Roboto Slab'
    code: 'Roboto Mono'
@ -16,6 +14,8 @@ extra:
      link: 'https://github.com/poseidon'
    - type: 'twitter'
      link: 'https://twitter.com/typhoon8s'
+repo_name: 'poseidon/typhoon'
+repo_url: 'https://github.com/poseidon/typhoon'
 google_analytics:
  - 'UA-38995133-6'
  - 'auto'
@ -52,6 +52,7 @@ pages:
    - 'Prometheus': 'addons/prometheus.md'
    - 'Dashboard': 'addons/dashboard.md'
  - 'Topics':
+    - 'Maintenance': 'topics/maintenance.md'
    - 'Hardware': 'topics/hardware.md'
    - 'Security': 'topics/security.md'
    - 'Performance': 'topics/performance.md'
--- a/requirements.txt
+++ b/requirements.txt
@ -1,4 +1,5 @@
-mkdocs==0.16.3
-mkdocs-material==1.8.0
+mkdocs==0.17.2
+mkdocs-material==2.2.6
 pygments==2.2.0
 pymdown-extensions==3.5
+six==1.10.0
Author	SHA1	Message	Date
Dalton Hubble	3dc755994b	Add missing changelog entry for Digital Ocean fix	2018-01-20 07:52:40 -08:00
Dalton Hubble	ddbfb2eee1	Set module version tutorials docs for good practice	2018-01-19 23:16:48 -08:00
Dalton Hubble	868265988b	Update bootkube and terraform-render-bootkube to v0.10.0	2018-01-19 23:10:45 -08:00
Dalton Hubble	6adffcb778	Update Kubernetes from v1.9.1 to v1.9.2	2018-01-19 08:40:09 -08:00
Dalton Hubble	bc967ddcd0	addons: Update CLUO to fix compatability with Kubernetes 1.9 * Update CLUO from v0.4.1 to v0.5.0 * Earlier versions of CLUO fail to drain nodes on Kubernetes 1.9 so nodes drain one at a time repeatedly and Container Linux OS updates are not applied to nodes. * Check current OS versions via `kubectl get nodes --show-labels`	2018-01-19 08:33:26 -08:00
Dalton Hubble	ef18f19ec4	Edit digital ocean port range and ordering to suppress diff * Change port range from keyword "all" to "1-65535", which is the same but with digitalocean provider 0.1.3 doesn't produce a diff * Rearrange egress firewall rules to order the Digtial Ocean API and provider returns. In current testing, this fixes the last diff that was present on `terraform plan`.	2018-01-15 22:13:59 -08:00
Dalton Hubble	f5efcc1ff8	Relax digitalocean provider version constraints * Relax fixed 0.1.2 version constraint to "~> 0.1.2", which allows 0.1.3, 0.1.4, etc, but would not allow 0.2.0	2018-01-15 21:04:53 -08:00
Dalton Hubble	996651c605	Update kube-state-metrics version and RBAC cluster role * https://github.com/kubernetes/kube-state-metrics/pull/345 * https://github.com/kubernetes/kube-state-metrics/pull/334	2018-01-15 08:33:44 -08:00
Dalton Hubble	38fa7dff1a	Create separate bare-metal container-linux-install profiles * Create separate container-linux-install profiles (and cached-container-linux-install) for each node in a cluster * Fix contention bug on bare-metal during `terraform apply`. With only a global install profile, terraform would create (or retain) the profile for each cluster and try to delete it for each cluster being deleted. As a result, in some cases apply had to be run multiple times before terraform's repr of constraints was satisfied (profile deleted and recreated) * Allow Container Linux install properties to vary between clusters, such as using a different Container Linux channel or version for different clusters	2018-01-15 08:23:03 -08:00
Dalton Hubble	bbe295a3f1	Add Terraform v0.11.x support and migration docs * Add explicit "providers" section to modules for Terraform v0.11.x * Retain support for Terraform v0.10.4+ * Add migration guide from Terraform v0.10.x to v0.11.x for those managing existing clusters (action required!)	2018-01-13 15:30:08 -08:00
Dalton Hubble	d8db296932	Update kube-dns and use separate service account * Update kube-dns from v1.14.7 to v1.14.8 * Use a separate kube-dns service account * https://github.com/kubernetes/kubernetes/pull/57918	2018-01-12 10:29:30 -08:00
Dalton Hubble	388ac08492	Update etcd from 3.2.13 to 3.2.14 * https://github.com/coreos/etcd/releases/tag/v3.2.14	2018-01-12 07:20:55 -08:00
Dalton Hubble	527b5ca602	Update CHANGELOG.md for v1.9.1	2018-01-09 07:03:04 -08:00
Dalton Hubble	ecd6a9443b	Add maintenance docs with upgrade policies * Add best practices for maintenance * Describe blue-green replacement strategy * Mention unsupported in-place edit and node replacement strategies	2018-01-09 06:54:44 -08:00
Dalton Hubble	2523d64f95	Fix docs to show exporting KUBECONFIG	2018-01-06 16:55:06 -08:00
Dalton Hubble	fc455c8624	Remove old mention of ACIs in bootkube.service description	2018-01-06 16:20:34 -08:00
Dalton Hubble	7a0a60708e	Bump Container Linux version shown in docs * Be sure docs and examples list Container Linux versions that have been patched for Meltdown just in case someone copy-pastes or sees them as recent versions	2018-01-06 14:58:38 -08:00
Dalton Hubble	51a5f64024	Enable portmap plugin alongside Calico to fix hostPort * https://github.com/poseidon/terraform-render-bootkube/pull/36	2018-01-06 14:01:18 -08:00
Dalton Hubble	e1f2125f02	Update etcd from 3.2.0 to 3.2.13 * https://github.com/coreos/etcd/releases/tag/v3.2.13	2018-01-06 14:01:18 -08:00
Dalton Hubble	9329b775f6	Update Kubernetes from v1.8.6 to v1.9.1	2018-01-06 14:01:16 -08:00
Dalton Hubble	e04cce1201	Update mkdocs and material docs theme	2018-01-06 10:59:56 -08:00