Update CHANGELOG.md for v1.8.6

Update Kubernetes from v1.8.5 to v1.8.6
Add link to dashboard 315
2025-08-03 12:31:33 +02:00 · 2017-12-22 13:00:18 -08:00 · 2017-12-21 11:20:37 -08:00 · 2017-12-15 18:52:40 -08:00 · 2017-12-15 16:09:14 -08:00 · 2017-12-15 02:04:13 -08:00
79 changed files with 1518 additions and 479 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,70 @@ Notable changes between versions.

 ## Latest

+## v1.8.6
+
+* Kubernetes [v1.8.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#v186)
+
+## v1.8.5
+
+* Kubernetes [v1.8.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#v185)
+* Recommend Container Linux [images](https://coreos.com/releases/) with Docker 17.09
+  * Container Linux stable, beta, and alpha now provide Docker 17.09 (instead
+  of 1.12)
+  * Older clusters (with CLUO addon) auto-update Container Linux version to begin using Docker 17.09
+* Fix race where `etcd-member.service` could fail to resolve peers ([#69](https://github.com/poseidon/typhoon/pull/69)) 
+* Add optional `cluster_domain_suffix` variable (#74)
+* Use kubernetes-incubator/bootkube v0.9.1
+
+#### Bare-Metal
+
+* Add kubelet `--volume-plugin-dir` flag to allow flexvolume providers ([#61](https://github.com/poseidon/typhoon/pull/61))
+
+#### Addons
+
+* Discourage deploying the Kubernetes Dashboard (security)
+
+## v1.8.4
+
+* Kubernetes v1.8.4
+* Calico related bug fixes
+* Update Calico from v2.6.1 to v2.6.3
+* Update flannel from v0.9.0 to v0.9.1
+* Service accounts for kube-proxy and pod-checkpointer
+* Use kubernetes-incubator/bootkube v0.9.0
+
+## v1.8.3
+
+* Kubernetes v1.8.3
+* Run etcd on-host, across controllers
+* Promote AWS platform to beta
+* Use kubernetes-incubator/bootkube v0.8.2
+
+#### Google Cloud
+
+* Add required variable `region` (e.g. "us-central1")
+* Reduce time to bootstrap a cluster
+* Change etcd to run on-host, across controllers (etcd-member.service)
+* Change controller instances to automatically span zones in the region
+* Change worker managed instance group to automatically span zones in the region
+* Improve internal firewall rules and use tag-based firewall policies
+* Remove support for self-hosted etcd
+* Remove the `zone` required variable
+* Remove the `controller_preemptible` optional variable
+
+#### AWS
+
+* Promote AWS platform to beta
+* Reduce time to bootstrap a cluster
+* Change etcd to run on-host, across controllers (etcd-member.service)
+* Fix firewall rules for multi-controller kubelet scraping and node-exporter
+* Remove support for self-hosted etcd
+
+#### Addons
+
+* Add Prometheus 2.0 addon with alerting rules
+* Add Grafana dashboard for observing metrics
+
 ## v1.8.2

 * Kubernetes v1.8.2
@ -54,7 +118,7 @@ Notable changes between versions.
 ## v1.7.3

 * Kubernetes v1.7.3
-* Use kubernete-incubator/bootkube v0.6.1
+* Use kubernetes-incubator/bootkube v0.6.1

 #### Digital Ocean

@ -64,7 +128,7 @@ Notable changes between versions.
 ## v1.7.1

 * Kubernetes v1.7.1
-* Use kubernete-incubator/bootkube v0.6.0
+* Use kubernetes-incubator/bootkube v0.6.0
 * Add Bare-Metal Terraform module (stable)
 * Add Digital Ocean Terraform module (beta)

@ -77,12 +141,12 @@ Notable changes between versions.
 ## v1.6.7

 * Kubernetes v1.6.7
-* Use kubernete-incubator/bootkube v0.5.1
+* Use kubernetes-incubator/bootkube v0.5.1

 ## v1.6.6

 * Kubernetes v1.6.6
-* Use kubernete-incubator/bootkube v0.4.5
+* Use kubernetes-incubator/bootkube v0.4.5
 * Disable locksmithd on hosts, in favor of [CLUO](https://github.com/coreos/container-linux-update-operator).

 ## v1.6.4
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@ -2,4 +2,4 @@

 ## Developer Certificate of Origin

-By contributing, you agree to the Linux Foundation's Developer Certificate of Origin ([DOC](DCO)). The DCO is a statement that you, the contributor, have the legal right to make your contribution and understand the contribution will be distributed as part of this project.
+By contributing, you agree to the Linux Foundation's Developer Certificate of Origin ([DCO](DCO)). The DCO is a statement that you, the contributor, have the legal right to make your contribution and understand the contribution will be distributed as part of this project.
--- a/README.md
+++ b/README.md
@ -1,4 +1,4 @@
-# Typhoon [![IRC](https://img.shields.io/badge/freenode-%23typhoon-0099ef.svg)]() <img align="right" src="https://storage.googleapis.com/dghubble/spin.png">
+# Typhoon [![IRC](https://img.shields.io/badge/freenode-%23typhoon-0099ef.svg)]() <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -9,9 +9,9 @@ Typhoon is a minimal and free Kubernetes distribution.

 Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.

-## Features
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
@ -22,7 +22,7 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | alpha |
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | beta |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | beta |
@ -46,10 +46,10 @@ module "google-cloud-yavin" {
  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"

  # Google Cloud
-  zone          = "us-central1-c"
+  region        = "us-central1"
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"
-  os_image      = "coreos-stable-1465-6-0-v20170817"
+  os_image      = "coreos-stable-1576-4-0-v20171206"

  cluster_name       = "yavin"
  controller_count   = 1
@ -72,15 +72,15 @@ $ terraform apply
 Apply complete! Resources: 37 added, 0 changed, 0 destroyed.
 ```

-In 5-10 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.
+In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.

 ```sh
 $ KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-1682.c.example-com.internal  Ready    6m     v1.8.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.8.6
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.6
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.6
 ```

 List the pods.
@ -91,13 +91,10 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
-kube-system   etcd-operator-3329263108-f443m            1/1    Running   1         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
 kube-system   kube-dns-1187388186-zj5dl                 3/3    Running   0         6m
-kube-system   kube-etcd-0000                            1/1    Running   0         5m
-kube-system   kube-etcd-network-checkpointer-crznb      1/1    Running   0         6m
 kube-system   kube-proxy-117v6                          1/1    Running   0         6m
 kube-system   kube-proxy-9886n                          1/1    Running   0         6m
 kube-system   kube-proxy-njn47                          1/1    Running   0         6m
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -0,0 +1,46 @@
+apiVersion: apps/v1beta2
+kind: Deployment
+metadata:
+  name: grafana
+  namespace: monitoring
+spec:
+  replicas: 1
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 1
+  selector:
+    matchLabels:
+      name: grafana
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: grafana
+        phase: prod
+    spec:
+      containers:
+        - name: grafana
+          image: grafana/grafana:4.6.3
+          env:
+            - name: GF_SERVER_HTTP_PORT
+              value: "8080"
+            - name: GF_AUTH_BASIC_ENABLED
+              value: "false"
+            - name: GF_AUTH_ANONYMOUS_ENABLED
+              value: "true"
+            - name: GF_AUTH_ANONYMOUS_ORG_ROLE
+              value: Admin
+          ports:
+            - name: http
+              containerPort: 8080
+          resources:
+            requests:
+              memory: 100Mi
+              cpu: 100m
+            limits:
+              memory: 200Mi
+              cpu: 200m
+      volumes:
+        - name: grafana-storage
+          emptyDir: {}
--- a/addons/grafana/service.yaml
+++ b/addons/grafana/service.yaml
@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: grafana
+  namespace: monitoring
+spec:
+  type: ClusterIP
+  selector:
+    name: grafana
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 8080
--- a/addons/heapster/deployment.yaml
+++ b/addons/heapster/deployment.yaml
@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1beta2
 kind: Deployment
 metadata:
  name: heapster
@ -19,7 +19,7 @@ spec:
    spec:
      containers:
        - name: heapster
-          image: gcr.io/google_containers/heapster-amd64:v1.4.3
+          image: gcr.io/google_containers/heapster-amd64:v1.5.0
          command:
            - /heapster
            - --source=kubernetes.summary_api:''
@ -31,16 +31,18 @@ spec:
            initialDelaySeconds: 180
            timeoutSeconds: 5
        - name: heapster-nanny
-          image: gcr.io/google_containers/addon-resizer:2.0
+          image: gcr.io/google_containers/addon-resizer:1.7
          command:
            - /pod_nanny
            - --cpu=80m
            - --extra-cpu=0.5m
            - --memory=140Mi
            - --extra-memory=4Mi
+            - --threshold=5
            - --deployment=heapster
            - --container=heapster
            - --poll-period=300000
+            - --estimator=exponential
          env:
            - name: MY_POD_NAME
              valueFrom:
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -19,7 +19,7 @@ spec:
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
-          image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -19,7 +19,7 @@ spec:
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
-          image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -19,7 +19,7 @@ spec:
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
-          image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
--- a/addons/prometheus/config.yaml
+++ b/addons/prometheus/config.yaml
@ -8,6 +8,14 @@ data:
    # Global config
    global:
      scrape_interval: 15s
+
+    # AlertManager
+    alerting:
+      alertmanagers:
+      - static_configs:
+        - targets:
+          - alertmanager:9093
+
    # Scrape configs for running Prometheus on a Kubernetes cluster.
    # This uses separate scrape configs for cluster components (i.e. API server, node)
    # and services to allow each to use different authentication configs.
@ -214,3 +222,5 @@ data:
    # Rule files
    rule_files:
      - "/etc/prometheus/rules/*.rules"
+      - "/etc/prometheus/rules/*.yaml"
+      - "/etc/prometheus/rules/*.yml"
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -16,11 +16,9 @@ spec:
    spec:
      containers:
      - name: prometheus
-        image: quay.io/prometheus/prometheus:v1.8.0
+        image: quay.io/prometheus/prometheus:v2.0.0
        args:
-          - '-config.file=/etc/prometheus/prometheus.yaml'
-          - '-storage.local.retention=12h'
-          - '-storage.local.memory-chunks=500000'
+          - '--config.file=/etc/prometheus/prometheus.yaml'
        ports:
        - name: web
          containerPort: 9090
--- a/addons/prometheus/discovery/kube-controller-manager.yaml
+++ b/addons/prometheus/discovery/kube-controller-manager.yaml
@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: kube-controller-manager
+  namespace: kube-system
+  annotations:
+    prometheus.io/scrape: 'true'
+spec:
+  type: ClusterIP
+  # service is created to allow prometheus to scrape endpoints
+  clusterIP: None
+  selector:
+    k8s-app: kube-controller-manager
+  ports:
+    - name: metrics
+      protocol: TCP
+      port: 10252
+      targetPort: 10252
--- a/addons/prometheus/discovery/kube-scheduler.yaml
+++ b/addons/prometheus/discovery/kube-scheduler.yaml
@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: kube-scheduler
+  namespace: kube-system
+  annotations:
+    prometheus.io/scrape: 'true'
+spec:
+  type: ClusterIP
+  # service is created to allow prometheus to scrape endpoints
+  clusterIP: None
+  selector:
+    k8s-app: kube-scheduler
+  ports:
+    - name: metrics
+      protocol: TCP
+      port: 10251
+      targetPort: 10251
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -6,8 +6,9 @@ metadata:
 spec:
  replicas: 1
  strategy:
+    type: RollingUpdate
    rollingUpdate:
-      maxUnavilable: 1
+      maxUnavailable: 1
  selector:
    matchLabels:
      name: kube-state-metrics
@ -31,13 +32,6 @@ spec:
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
-        resources:
-          requests:
-            memory: 100Mi
-            cpu: 100m
-          limits:
-            memory: 200Mi
-            cpu: 200m
      - name: addon-resizer
        image: gcr.io/google_containers/addon-resizer:1.0
        resources:
--- a/addons/prometheus/exporters/kube-state-metrics/service.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/service.yaml
@ -7,6 +7,8 @@ metadata:
    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
+  # service is created to allow prometheus to scape endpoints
+  clusterIP: None
  selector:
    name: kube-state-metrics
    phase: prod
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -4,5 +4,543 @@ metadata:
  name: prometheus-rules
  namespace: monitoring
 data:
-  example.rule: |
-    job_service:rpc_durations_seconds_count:avg_rate5m = avg(rate(rpc_durations_seconds_count[5m])) by (job, service)
+  # Rules adapted from those provided by coreos/prometheus-operator and SoundCloud
+  alertmanager.rules.yaml: |+
+    groups:
+    - name: alertmanager.rules
+      rules:
+      - alert: AlertmanagerConfigInconsistent
+        expr: count_values("config_hash", alertmanager_config_hash) BY (service) / ON(service)
+          GROUP_LEFT() label_replace(prometheus_operator_alertmanager_spec_replicas, "service",
+          "alertmanager-$1", "alertmanager", "(.*)") != 1
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          description: The configuration of the instances of the Alertmanager cluster
+            `{{$labels.service}}` are out of sync.
+      - alert: AlertmanagerDownOrMissing
+        expr: label_replace(prometheus_operator_alertmanager_spec_replicas, "job", "alertmanager-$1",
+          "alertmanager", "(.*)") / ON(job) GROUP_RIGHT() sum(up) BY (job) != 1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          description: An unexpected number of Alertmanagers are scraped or Alertmanagers
+            disappeared from discovery.
+      - alert: AlertmanagerFailedReload
+        expr: alertmanager_config_last_reload_successful == 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Reloading Alertmanager's configuration has failed for {{ $labels.namespace
+            }}/{{ $labels.pod}}.
+  etcd3.rules.yaml: |+
+    groups:
+    - name: ./etcd3.rules
+      rules:
+      - alert: InsufficientMembers
+        expr: count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2 - 1)
+        for: 3m
+        labels:
+          severity: critical
+        annotations:
+          description: If one more etcd member goes down the cluster will be unavailable
+          summary: etcd cluster insufficient members
+      - alert: NoLeader
+        expr: etcd_server_has_leader{job="etcd"} == 0
+        for: 1m
+        labels:
+          severity: critical
+        annotations:
+          description: etcd member {{ $labels.instance }} has no leader
+          summary: etcd member has no leader
+      - alert: HighNumberOfLeaderChanges
+        expr: increase(etcd_server_leader_changes_seen_total{job="etcd"}[1h]) > 3
+        labels:
+          severity: warning
+        annotations:
+          description: etcd instance {{ $labels.instance }} has seen {{ $value }} leader
+            changes within the last hour
+          summary: a high number of leader changes within the etcd cluster are happening
+      - alert: HighNumberOfFailedGRPCRequests
+        expr: sum(rate(etcd_grpc_requests_failed_total{job="etcd"}[5m])) BY (grpc_method)
+          / sum(rate(etcd_grpc_total{job="etcd"}[5m])) BY (grpc_method) > 0.01
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: '{{ $value }}% of requests for {{ $labels.grpc_method }} failed
+            on etcd instance {{ $labels.instance }}'
+          summary: a high number of gRPC requests are failing
+      - alert: HighNumberOfFailedGRPCRequests
+        expr: sum(rate(etcd_grpc_requests_failed_total{job="etcd"}[5m])) BY (grpc_method)
+          / sum(rate(etcd_grpc_total{job="etcd"}[5m])) BY (grpc_method) > 0.05
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          description: '{{ $value }}% of requests for {{ $labels.grpc_method }} failed
+            on etcd instance {{ $labels.instance }}'
+          summary: a high number of gRPC requests are failing
+      - alert: GRPCRequestsSlow
+        expr: histogram_quantile(0.99, rate(etcd_grpc_unary_requests_duration_seconds_bucket[5m]))
+          > 0.15
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          description: on etcd instance {{ $labels.instance }} gRPC requests to {{ $labels.grpc_method
+            }} are slow
+          summary: slow gRPC requests
+      - alert: HighNumberOfFailedHTTPRequests
+        expr: sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job="etcd"}[5m]))
+          BY (method) > 0.01
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
+            instance {{ $labels.instance }}'
+          summary: a high number of HTTP requests are failing
+      - alert: HighNumberOfFailedHTTPRequests
+        expr: sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job="etcd"}[5m]))
+          BY (method) > 0.05
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          description: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
+            instance {{ $labels.instance }}'
+          summary: a high number of HTTP requests are failing
+      - alert: HTTPRequestsSlow
+        expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))
+          > 0.15
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: on etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method
+            }} are slow
+          summary: slow HTTP requests
+      - alert: EtcdMemberCommunicationSlow
+        expr: histogram_quantile(0.99, rate(etcd_network_member_round_trip_time_seconds_bucket[5m]))
+          > 0.15
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: etcd instance {{ $labels.instance }} member communication with
+            {{ $labels.To }} is slow
+          summary: etcd member communication is slow
+      - alert: HighNumberOfFailedProposals
+        expr: increase(etcd_server_proposals_failed_total{job="etcd"}[1h]) > 5
+        labels:
+          severity: warning
+        annotations:
+          description: etcd instance {{ $labels.instance }} has seen {{ $value }} proposal
+            failures within the last hour
+          summary: a high number of proposals within the etcd cluster are failing
+      - alert: HighFsyncDurations
+        expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))
+          > 0.5
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: etcd instance {{ $labels.instance }} fync durations are high
+          summary: high fsync durations
+      - alert: HighCommitDurations
+        expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))
+          > 0.25
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: etcd instance {{ $labels.instance }} commit durations are high
+          summary: high commit durations
+  general.rules.yaml: |+
+    groups:
+    - name: general.rules
+      rules:
+      - alert: TargetDown
+        expr: 100 * (count(up == 0) BY (job) / count(up) BY (job)) > 10
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: '{{ $value }}% of {{ $labels.job }} targets are down.'
+          summary: Targets are down
+      - record: fd_utilization
+        expr: process_open_fds / process_max_fds
+      - alert: FdExhaustionClose
+        expr: predict_linear(fd_utilization[1h], 3600 * 4) > 1
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: '{{ $labels.job }}: {{ $labels.namespace }}/{{ $labels.pod }} instance
+            will exhaust in file/socket descriptors within the next 4 hours'
+          summary: file descriptors soon exhausted
+      - alert: FdExhaustionClose
+        expr: predict_linear(fd_utilization[10m], 3600) > 1
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          description: '{{ $labels.job }}: {{ $labels.namespace }}/{{ $labels.pod }} instance
+            will exhaust in file/socket descriptors within the next hour'
+          summary: file descriptors soon exhausted
+  kube-controller-manager.rules.yaml: |+
+    groups:
+    - name: kube-controller-manager.rules
+      rules:
+      - alert: K8SControllerManagerDown
+        expr: absent(up{kubernetes_name="kube-controller-manager"} == 1)
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          description: There is no running K8S controller manager. Deployments and replication
+            controllers are not making progress.
+          summary: Controller manager is down
+  kube-scheduler.rules.yaml: |+
+    groups:
+    - name: kube-scheduler.rules
+      rules:
+      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
+        expr: histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.99"
+      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
+        expr: histogram_quantile(0.9, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.9"
+      - record: cluster:scheduler_e2e_scheduling_latency_seconds:quantile
+        expr: histogram_quantile(0.5, sum(scheduler_e2e_scheduling_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.5"
+      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
+        expr: histogram_quantile(0.99, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.99"
+      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
+        expr: histogram_quantile(0.9, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.9"
+      - record: cluster:scheduler_scheduling_algorithm_latency_seconds:quantile
+        expr: histogram_quantile(0.5, sum(scheduler_scheduling_algorithm_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.5"
+      - record: cluster:scheduler_binding_latency_seconds:quantile
+        expr: histogram_quantile(0.99, sum(scheduler_binding_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.99"
+      - record: cluster:scheduler_binding_latency_seconds:quantile
+        expr: histogram_quantile(0.9, sum(scheduler_binding_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.9"
+      - record: cluster:scheduler_binding_latency_seconds:quantile
+        expr: histogram_quantile(0.5, sum(scheduler_binding_latency_microseconds_bucket)
+          BY (le, cluster)) / 1e+06
+        labels:
+          quantile: "0.5"
+      - alert: K8SSchedulerDown
+        expr: absent(up{kubernetes_name="kube-scheduler"} == 1)
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          description: There is no running K8S scheduler. New pods are not being assigned
+            to nodes.
+          summary: Scheduler is down
+  kube-state-metrics.rules.yaml: |+
+    groups:
+    - name: kube-state-metrics.rules
+      rules:
+      - alert: DeploymentGenerationMismatch
+        expr: kube_deployment_status_observed_generation != kube_deployment_metadata_generation
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          description: Observed deployment generation does not match expected one for
+            deployment {{$labels.namespaces}}{{$labels.deployment}}
+      - alert: DeploymentReplicasNotUpdated
+        expr: ((kube_deployment_status_replicas_updated != kube_deployment_spec_replicas)
+          or (kube_deployment_status_replicas_available != kube_deployment_spec_replicas))
+          unless (kube_deployment_spec_paused == 1)
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          description: Replicas are not updated and available for deployment {{$labels.namespaces}}/{{$labels.deployment}}
+      - alert: DaemonSetRolloutStuck
+        expr: kube_daemonset_status_current_number_ready / kube_daemonset_status_desired_number_scheduled
+          * 100 < 100
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          description: Only {{$value}}% of desired pods scheduled and ready for daemon
+            set {{$labels.namespaces}}/{{$labels.daemonset}}
+      - alert: K8SDaemonSetsNotScheduled
+        expr: kube_daemonset_status_desired_number_scheduled - kube_daemonset_status_current_number_scheduled
+          > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: A number of daemonsets are not scheduled.
+          summary: Daemonsets are not scheduled correctly
+      - alert: DaemonSetsMissScheduled
+        expr: kube_daemonset_status_number_misscheduled > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: A number of daemonsets are running where they are not supposed
+            to run.
+          summary: Daemonsets are not scheduled correctly
+      - alert: PodFrequentlyRestarting
+        expr: increase(kube_pod_container_status_restarts[1h]) > 5
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Pod {{$labels.namespaces}}/{{$labels.pod}} is was restarted {{$value}}
+            times within the last hour
+  kubelet.rules.yaml: |+
+    groups:
+    - name: kubelet.rules
+      rules:
+      - alert: K8SNodeNotReady
+        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
+        for: 1h
+        labels:
+          severity: warning
+        annotations:
+          description: The Kubelet on {{ $labels.node }} has not checked in with the API,
+            or has set itself to NotReady, for more than an hour
+          summary: Node status is NotReady
+      - alert: K8SManyNodesNotReady
+        expr: count(kube_node_status_condition{condition="Ready",status="true"} == 0)
+          > 1 and (count(kube_node_status_condition{condition="Ready",status="true"} ==
+          0) / count(kube_node_status_condition{condition="Ready",status="true"})) > 0.2
+        for: 1m
+        labels:
+          severity: critical
+        annotations:
+          description: '{{ $value }}% of Kubernetes nodes are not ready'
+      - alert: K8SKubeletDown
+        expr: count(up{job="kubernetes-nodes"} == 0) / count(up{job="kubernetes-nodes"}) * 100 > 3
+        for: 1h
+        labels:
+          severity: warning
+        annotations:
+          description: Prometheus failed to scrape {{ $value }}% of kubelets.
+      - alert: K8SKubeletDown
+        expr: (absent(up{job="kubernetes-nodes"} == 1) or count(up{job="kubernetes-nodes"} == 0) / count(up{job="kubernetes-nodes"}))
+          * 100 > 1
+        for: 1h
+        labels:
+          severity: critical
+        annotations:
+          description: Prometheus failed to scrape {{ $value }}% of kubelets, or all Kubelets
+            have disappeared from service discovery.
+          summary: Many Kubelets cannot be scraped
+      - alert: K8SKubeletTooManyPods
+        expr: kubelet_running_pod_count > 100
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Kubelet {{$labels.instance}} is running {{$value}} pods, close
+            to the limit of 110
+          summary: Kubelet is close to pod limit
+  kubernetes.rules.yaml: |+
+    groups:
+    - name: kubernetes.rules
+      rules:
+      - record: pod_name:container_memory_usage_bytes:sum
+        expr: sum(container_memory_usage_bytes{container_name!="POD",pod_name!=""}) BY
+          (pod_name)
+      - record: pod_name:container_spec_cpu_shares:sum
+        expr: sum(container_spec_cpu_shares{container_name!="POD",pod_name!=""}) BY (pod_name)
+      - record: pod_name:container_cpu_usage:sum
+        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
+          BY (pod_name)
+      - record: pod_name:container_fs_usage_bytes:sum
+        expr: sum(container_fs_usage_bytes{container_name!="POD",pod_name!=""}) BY (pod_name)
+      - record: namespace:container_memory_usage_bytes:sum
+        expr: sum(container_memory_usage_bytes{container_name!=""}) BY (namespace)
+      - record: namespace:container_spec_cpu_shares:sum
+        expr: sum(container_spec_cpu_shares{container_name!=""}) BY (namespace)
+      - record: namespace:container_cpu_usage:sum
+        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD"}[5m]))
+          BY (namespace)
+      - record: cluster:memory_usage:ratio
+        expr: sum(container_memory_usage_bytes{container_name!="POD",pod_name!=""}) BY
+          (cluster) / sum(machine_memory_bytes) BY (cluster)
+      - record: cluster:container_spec_cpu_shares:ratio
+        expr: sum(container_spec_cpu_shares{container_name!="POD",pod_name!=""}) / 1000
+          / sum(machine_cpu_cores)
+      - record: cluster:container_cpu_usage:ratio
+        expr: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
+          / sum(machine_cpu_cores)
+      - record: apiserver_latency_seconds:quantile
+        expr: histogram_quantile(0.99, rate(apiserver_request_latencies_bucket[5m])) /
+          1e+06
+        labels:
+          quantile: "0.99"
+      - record: apiserver_latency:quantile_seconds
+        expr: histogram_quantile(0.9, rate(apiserver_request_latencies_bucket[5m])) /
+          1e+06
+        labels:
+          quantile: "0.9"
+      - record: apiserver_latency_seconds:quantile
+        expr: histogram_quantile(0.5, rate(apiserver_request_latencies_bucket[5m])) /
+          1e+06
+        labels:
+          quantile: "0.5"
+      - alert: APIServerLatencyHigh
+        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
+          > 1
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: the API server has a 99th percentile latency of {{ $value }} seconds
+            for {{$labels.verb}} {{$labels.resource}}
+      - alert: APIServerLatencyHigh
+        expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
+          > 4
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          description: the API server has a 99th percentile latency of {{ $value }} seconds
+            for {{$labels.verb}} {{$labels.resource}}
+      - alert: APIServerErrorsHigh
+        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m])
+          * 100 > 2
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: API server returns errors for {{ $value }}% of requests
+      - alert: APIServerErrorsHigh
+        expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m]) / rate(apiserver_request_count[5m])
+          * 100 > 5
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          description: API server returns errors for {{ $value }}% of requests
+      - alert: K8SApiserverDown
+        expr: absent(up{job="kubernetes-apiservers"} == 1)
+        for: 20m
+        labels:
+          severity: critical
+        annotations:
+          description: No API servers are reachable or all have disappeared from service
+            discovery
+  node.rules.yaml: |+
+    groups:
+    - name: node.rules
+      rules:
+      - record: instance:node_cpu:rate:sum
+        expr: sum(rate(node_cpu{mode!="idle",mode!="iowait",mode!~"^(?:guest.*)$"}[3m]))
+          BY (instance)
+      - record: instance:node_filesystem_usage:sum
+        expr: sum((node_filesystem_size{mountpoint="/"} - node_filesystem_free{mountpoint="/"}))
+          BY (instance)
+      - record: instance:node_network_receive_bytes:rate:sum
+        expr: sum(rate(node_network_receive_bytes[3m])) BY (instance)
+      - record: instance:node_network_transmit_bytes:rate:sum
+        expr: sum(rate(node_network_transmit_bytes[3m])) BY (instance)
+      - record: instance:node_cpu:ratio
+        expr: sum(rate(node_cpu{mode!="idle"}[5m])) WITHOUT (cpu, mode) / ON(instance)
+          GROUP_LEFT() count(sum(node_cpu) BY (instance, cpu)) BY (instance)
+      - record: cluster:node_cpu:sum_rate5m
+        expr: sum(rate(node_cpu{mode!="idle"}[5m]))
+      - record: cluster:node_cpu:ratio
+        expr: cluster:node_cpu:rate5m / count(sum(node_cpu) BY (instance, cpu))
+      - alert: NodeExporterDown
+        expr: absent(up{kubernetes_name="node-exporter"} == 1)
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Prometheus could not scrape a node-exporter for more than 10m,
+            or node-exporters have disappeared from discovery
+      - alert: NodeDiskRunningFull
+        expr: predict_linear(node_filesystem_free[6h], 3600 * 24) < 0
+        for: 30m
+        labels:
+          severity: warning
+        annotations:
+          description: device {{$labels.device}} on node {{$labels.instance}} is running
+            full within the next 24 hours (mounted at {{$labels.mountpoint}})
+      - alert: NodeDiskRunningFull
+        expr: predict_linear(node_filesystem_free[30m], 3600 * 2) < 0
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          description: device {{$labels.device}} on node {{$labels.instance}} is running
+            full within the next 2 hours (mounted at {{$labels.mountpoint}})
+  prometheus.rules.yaml: |+
+    groups:
+    - name: prometheus.rules
+      rules:
+      - alert: PrometheusConfigReloadFailed
+        expr: prometheus_config_last_reload_successful == 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{$labels.pod}}
+      - alert: PrometheusNotificationQueueRunningFull
+        expr: predict_linear(prometheus_notifications_queue_length[5m], 60 * 30) > prometheus_notifications_queue_capacity
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Prometheus' alert notification queue is running full for {{$labels.namespace}}/{{
+            $labels.pod}}
+      - alert: PrometheusErrorSendingAlerts
+        expr: rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m])
+          > 0.01
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Errors while sending alerts from Prometheus {{$labels.namespace}}/{{
+            $labels.pod}} to Alertmanager {{$labels.Alertmanager}}
+      - alert: PrometheusErrorSendingAlerts
+        expr: rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m])
+          > 0.03
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          description: Errors while sending alerts from Prometheus {{$labels.namespace}}/{{
+            $labels.pod}} to Alertmanager {{$labels.Alertmanager}}
+      - alert: PrometheusNotConnectedToAlertmanagers
+        expr: prometheus_notifications_alertmanagers_discovered < 1
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: Prometheus {{ $labels.namespace }}/{{ $labels.pod}} is not connected
+            to any Alertmanagers
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -1,4 +1,4 @@
-# Typhoon
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -9,9 +9,9 @@ Typhoon is a minimal and free Kubernetes distribution.

 Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.

-## Features
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/aws/container-linux/kubernetes/bootkube.tf
+++ b/aws/container-linux/kubernetes/bootkube.tf
@ -1,14 +1,14 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.8.1"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"

-  cluster_name                  = "${var.cluster_name}"
-  api_servers                   = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers                  = ["http://127.0.0.1:2379"]
-  asset_dir                     = "${var.asset_dir}"
-  networking                    = "${var.networking}"
-  network_mtu                   = "${var.network_mtu}"
-  pod_cidr                      = "${var.pod_cidr}"
-  service_cidr                  = "${var.service_cidr}"
-  experimental_self_hosted_etcd = "true"
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  etcd_servers          = ["${aws_route53_record.etcds.*.fqdn}"]
+  asset_dir             = "${var.asset_dir}"
+  networking            = "${var.networking}"
+  network_mtu           = "${var.network_mtu}"
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
 }
--- a/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -1,6 +1,29 @@
 ---
 systemd:
  units:
+    - name: etcd-member.service
+      enable: true
+      dropins:
+        - name: 40-etcd-cluster.conf
+          contents: |
+            [Service]
+            Environment="ETCD_IMAGE_TAG=v3.2.0"
+            Environment="ETCD_NAME=${etcd_name}"
+            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
+            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
+            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
+            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
+            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
+            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
+            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
+            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
+            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
+            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
+            Environment="ETCD_CLIENT_CERT_AUTH=true"
+            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
+            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
+            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
+            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: docker.service
      enable: true
    - name: locksmithd.service
@ -18,6 +41,7 @@ systemd:
        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
        [Install]
        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
    - name: kubelet.service
      enable: true
      contents: |
@ -49,7 +73,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --kubeconfig=/etc/kubernetes/kubeconfig \
@ -57,8 +81,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/master \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-          --require-kubeconfig
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=10
@ -106,7 +129,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -125,11 +148,9 @@ storage:
          # Wrapper for bootkube start
          set -e
          # Move experimental manifests
-          [ -d /opt/bootkube/assets/manifests-* ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          [ -d /opt/bootkube/assets/experimental/manifests ] && mv /opt/bootkube/assets/experimental/manifests/* /opt/bootkube/assets/manifests && rm -r /opt/bootkube/assets/experimental/manifests
-          [ -d /opt/bootkube/assets/experimental/bootstrap-manifests ] && mv /opt/bootkube/assets/experimental/bootstrap-manifests/* /opt/bootkube/assets/bootstrap-manifests && rm -r /opt/bootkube/assets/experimental/bootstrap-manifests
+          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.8.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/aws/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -49,15 +49,14 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --require-kubeconfig
+          --pod-manifest-path=/etc/kubernetes/manifests
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=5
@ -104,7 +103,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -122,7 +121,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.8.2 \
+            docker://gcr.io/google_containers/hyperkube:v1.8.6 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -1,39 +1,30 @@
-# Controllers AutoScaling Group
-resource "aws_autoscaling_group" "controllers" {
-  name           = "${var.cluster_name}-controller"
-  load_balancers = ["${aws_elb.controllers.id}"]
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "aws_route53_record" "etcds" {
+  count = "${var.controller_count}"

-  # count
-  desired_capacity = "${var.controller_count}"
-  min_size         = "${var.controller_count}"
-  max_size         = "${var.controller_count}"
+  # DNS Zone where record should be created
+  zone_id = "${var.dns_zone_id}"

-  # network
-  vpc_zone_identifier = ["${aws_subnet.public.*.id}"]
+  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)}"
+  type = "A"
+  ttl  = 300

-  # template
-  launch_configuration = "${aws_launch_configuration.controller.name}"
-
-  lifecycle {
-    # override the default destroy and replace update behavior
-    create_before_destroy = true
-    ignore_changes        = ["image_id"]
-  }
-
-  tags = [{
-    key                 = "Name"
-    value               = "${var.cluster_name}-controller"
-    propagate_at_launch = true
-  }]
+  # private IPv4 address for etcd
+  records = ["${element(aws_instance.controllers.*.private_ip, count.index)}"]
 }

-# Controller template
-resource "aws_launch_configuration" "controller" {
-  name_prefix   = "${var.cluster_name}-controller-template-"
-  image_id      = "${data.aws_ami.coreos.image_id}"
+# Controller instances
+resource "aws_instance" "controllers" {
+  count = "${var.controller_count}"
+
+  tags = {
+    Name = "${var.cluster_name}-controller-${count.index}"
+  }
+
  instance_type = "${var.controller_type}"

-  user_data = "${data.ct_config.controller_ign.rendered}"
+  ami       = "${data.aws_ami.coreos.image_id}"
+  user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"

  # storage
  root_block_device {
@ -43,22 +34,27 @@ resource "aws_launch_configuration" "controller" {

  # network
  associate_public_ip_address = true
-  security_groups             = ["${aws_security_group.controller.id}"]
-
-  lifecycle {
-    // Override the default destroy and replace update behavior
-    create_before_destroy = true
-  }
+  subnet_id                   = "${element(aws_subnet.public.*.id, count.index)}"
+  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
 }

 # Controller Container Linux Config
 data "template_file" "controller_config" {
+  count = "${var.controller_count}"
+
  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"

  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster = "${join(",", formatlist("%s=https://%s:2380", null_resource.repeat.*.triggers.name, null_resource.repeat.*.triggers.domain))}"
+
    k8s_dns_service_ip      = "${cidrhost(var.service_cidr, 10)}"
-    k8s_etcd_service_ip     = "${cidrhost(var.service_cidr, 15)}"
    ssh_authorized_key      = "${var.ssh_authorized_key}"
+    cluster_domain_suffix   = "${var.cluster_domain_suffix}"
    kubeconfig_ca_cert      = "${module.bootkube.ca_cert}"
    kubeconfig_kubelet_cert = "${module.bootkube.kubelet_cert}"
    kubeconfig_kubelet_key  = "${module.bootkube.kubelet_key}"
@ -66,8 +62,20 @@ data "template_file" "controller_config" {
  }
 }

+# Horrible hack to generate a Terraform list of a desired length without dependencies.
+# Ideal ${repeat("etcd", 3) -> ["etcd", "etcd", "etcd"]}
+resource null_resource "repeat" {
+  count = "${var.controller_count}"
+
+  triggers {
+    name   = "etcd${count.index}"
+    domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+  }
+}
+
 data "ct_config" "controller_ign" {
-  content      = "${data.template_file.controller_config.rendered}"
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
  pretty_print = false
 }

@ -122,16 +130,6 @@ resource "aws_security_group_rule" "controller-etcd" {
  self      = true
 }

-resource "aws_security_group_rule" "controller-bootstrap-etcd" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 12379
-  to_port   = 12380
-  self      = true
-}
-
 resource "aws_security_group_rule" "controller-flannel" {
  security_group_id = "${aws_security_group.controller.id}"

@ -152,6 +150,26 @@ resource "aws_security_group_rule" "controller-flannel-self" {
  self      = true
 }

+resource "aws_security_group_rule" "controller-node-exporter" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 9100
+  to_port                  = 9100
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-kubelet-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10250
+  to_port   = 10250
+  self      = true
+}
+
 resource "aws_security_group_rule" "controller-kubelet-read" {
  security_group_id = "${aws_security_group.controller.id}"

--- a/aws/container-linux/kubernetes/elb.tf
+++ b/aws/container-linux/kubernetes/elb.tf
@ -1,5 +1,5 @@
-# Controller Network Load Balancer DNS Record
-resource "aws_route53_record" "controllers" {
+# kube-apiserver Network Load Balancer DNS Record
+resource "aws_route53_record" "apiserver" {
  zone_id = "${var.dns_zone_id}"

  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
@ -7,25 +7,18 @@ resource "aws_route53_record" "controllers" {

  # AWS recommends their special "alias" records for ELBs
  alias {
-    name                   = "${aws_elb.controllers.dns_name}"
-    zone_id                = "${aws_elb.controllers.zone_id}"
+    name                   = "${aws_elb.apiserver.dns_name}"
+    zone_id                = "${aws_elb.apiserver.zone_id}"
    evaluate_target_health = true
  }
 }

 # Controller Network Load Balancer
-resource "aws_elb" "controllers" {
-  name            = "${var.cluster_name}-controllers"
+resource "aws_elb" "apiserver" {
+  name            = "${var.cluster_name}-apiserver"
  subnets         = ["${aws_subnet.public.*.id}"]
  security_groups = ["${aws_security_group.controller.id}"]

-  listener {
-    lb_port           = 22
-    lb_protocol       = "tcp"
-    instance_port     = 22
-    instance_protocol = "tcp"
-  }
-
  listener {
    lb_port           = 443
    lb_protocol       = "tcp"
@ -33,16 +26,18 @@ resource "aws_elb" "controllers" {
    instance_protocol = "tcp"
  }

+  instances = ["${aws_instance.controllers.*.id}"]
+
  # Kubelet HTTP health check
  health_check {
-    target              = "HTTP:10255/healthz"
+    target              = "SSL:443"
    healthy_threshold   = 2
    unhealthy_threshold = 4
    timeout             = 5
    interval            = 6
  }

-  idle_timeout                = 1800
+  idle_timeout                = 3600
  connection_draining         = true
  connection_draining_timeout = 300
 }
--- a/aws/container-linux/kubernetes/ssh.tf
+++ b/aws/container-linux/kubernetes/ssh.tf
@ -1,12 +1,79 @@
+# Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
+resource "null_resource" "copy-secrets" {
+  count = "${var.controller_count}"
+
+  connection {
+    type    = "ssh"
+    host    = "${element(aws_instance.controllers.*.public_ip, count.index)}"
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.kubeconfig}"
+    destination = "$HOME/kubeconfig"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_ca_cert}"
+    destination = "$HOME/etcd-client-ca.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_cert}"
+    destination = "$HOME/etcd-client.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_key}"
+    destination = "$HOME/etcd-client.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_cert}"
+    destination = "$HOME/etcd-server.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_key}"
+    destination = "$HOME/etcd-server.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_cert}"
+    destination = "$HOME/etcd-peer.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_key}"
+    destination = "$HOME/etcd-peer.key"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mkdir -p /etc/ssl/etcd/etcd",
+      "sudo mv etcd-client* /etc/ssl/etcd/",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
+      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
+      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
+      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
+      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
+      "sudo chown -R etcd:etcd /etc/ssl/etcd",
+      "sudo chmod -R 500 /etc/ssl/etcd",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
+    ]
+  }
+}
+
 # Secure copy bootkube assets to ONE controller and start bootkube to perform
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
-  depends_on = ["module.bootkube", "aws_autoscaling_group.controllers"]
+  depends_on = ["module.bootkube", "null_resource.copy-secrets", "aws_route53_record.apiserver"]

-  # TODO: SSH to a controller's IP instead of waiting on DNS resolution
  connection {
    type    = "ssh"
-    host    = "${aws_route53_record.controllers.fqdn}"
+    host    = "${aws_instance.controllers.0.public_ip}"
    user    = "core"
    timeout = "15m"
  }
--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -88,9 +88,15 @@ variable "pod_cidr" {
 variable "service_cidr" {
  description = <<EOD
 CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns, the 15th IP will be reserved for self-hosted etcd, and the 200th IP will be reserved for bootstrap self-hosted etcd.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
 EOD

  type    = "string"
  default = "10.3.0.0/16"
 }
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
--- a/aws/container-linux/kubernetes/workers.tf
+++ b/aws/container-linux/kubernetes/workers.tf
@ -59,6 +59,7 @@ data "template_file" "worker_config" {
    k8s_dns_service_ip      = "${cidrhost(var.service_cidr, 10)}"
    k8s_etcd_service_ip     = "${cidrhost(var.service_cidr, 15)}"
    ssh_authorized_key      = "${var.ssh_authorized_key}"
+    cluster_domain_suffix   = "${var.cluster_domain_suffix}"
    kubeconfig_ca_cert      = "${module.bootkube.ca_cert}"
    kubeconfig_kubelet_cert = "${module.bootkube.kubelet_cert}"
    kubeconfig_kubelet_key  = "${module.bootkube.kubelet_key}"
@ -142,6 +143,16 @@ resource "aws_security_group_rule" "worker-flannel-self" {
  self      = true
 }

+resource "aws_security_group_rule" "worker-node-exporter" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 9100
+  to_port     = 9100
+  self = true
+}
+
 resource "aws_security_group_rule" "worker-kubelet" {
  security_group_id = "${aws_security_group.worker.id}"

--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -1,4 +1,4 @@
-# Typhoon
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -9,9 +9,9 @@ Typhoon is a minimal and free Kubernetes distribution.

 Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.

-## Features
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/bare-metal/container-linux/kubernetes/bootkube.tf
+++ b/bare-metal/container-linux/kubernetes/bootkube.tf
@ -1,13 +1,14 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.8.1"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"

-  cluster_name = "${var.cluster_name}"
-  api_servers  = ["${var.k8s_domain_name}"]
-  etcd_servers = ["${var.controller_domains}"]
-  asset_dir    = "${var.asset_dir}"
-  networking   = "${var.networking}"
-  network_mtu  = "${var.network_mtu}"
-  pod_cidr     = "${var.pod_cidr}"
-  service_cidr = "${var.service_cidr}"
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${var.k8s_domain_name}"]
+  etcd_servers          = ["${var.controller_domains}"]
+  asset_dir             = "${var.asset_dir}"
+  networking            = "${var.networking}"
+  network_mtu           = "${var.network_mtu}"
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
 }
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -50,6 +50,7 @@ systemd:
        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
        [Install]
        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
    - name: kubelet.service
      contents: |
        [Unit]
@ -73,6 +74,7 @@ systemd:
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
        ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
        ExecStart=/usr/lib/coreos/kubelet-wrapper \
@ -80,7 +82,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override=${domain_name} \
@ -90,7 +92,7 @@ systemd:
          --node-labels=node-role.kubernetes.io/master \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-          --require-kubeconfig
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=10
@ -115,7 +117,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/hostname
      filesystem: root
      mode: 0644
@ -140,11 +142,9 @@ storage:
          # Wrapper for bootkube start
          set -e
          # Move experimental manifests
-          [ -d /opt/bootkube/assets/manifests-* ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          [ -d /opt/bootkube/assets/experimental/manifests ] && mv /opt/bootkube/assets/experimental/manifests/* /opt/bootkube/assets/manifests && rm -r /opt/bootkube/assets/experimental/manifests
-          [ -d /opt/bootkube/assets/experimental/bootstrap-manifests ] && mv /opt/bootkube/assets/experimental/bootstrap-manifests/* /opt/bootkube/assets/bootstrap-manifests && rm -r /opt/bootkube/assets/experimental/bootstrap-manifests
+          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.8.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -50,6 +50,7 @@ systemd:
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
        ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
        ExecStart=/usr/lib/coreos/kubelet-wrapper \
@ -57,7 +58,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override=${domain_name} \
@ -66,7 +67,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --require-kubeconfig
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=5
@ -81,7 +82,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -8,6 +8,7 @@ resource "matchbox_profile" "container-linux-install" {
  ]

  args = [
+    "initrd=coreos_production_pxe_image.cpio.gz",
    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.first_boot=yes",
    "console=tty0",
@ -44,6 +45,7 @@ resource "matchbox_profile" "cached-container-linux-install" {
  ]

  args = [
+    "initrd=coreos_production_pxe_image.cpio.gz",
    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.first_boot=yes",
    "console=tty0",
@ -82,11 +84,12 @@ data "template_file" "controller-configs" {
  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"

  vars {
-    domain_name          = "${element(var.controller_domains, count.index)}"
-    etcd_name            = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    k8s_dns_service_ip   = "${module.bootkube.kube_dns_service_ip}"
-    ssh_authorized_key   = "${var.ssh_authorized_key}"
+    domain_name           = "${element(var.controller_domains, count.index)}"
+    etcd_name             = "${element(var.controller_names, count.index)}"
+    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
+    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"

    # Terraform evaluates both sides regardless and element cannot be used on 0 length lists
    networkd_content = "${length(var.controller_networkds) == 0 ? "" : element(concat(var.controller_networkds, list("")), count.index)}"
@ -106,9 +109,10 @@ data "template_file" "worker-configs" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars {
-    domain_name        = "${element(var.worker_domains, count.index)}"
-    k8s_dns_service_ip = "${module.bootkube.kube_dns_service_ip}"
-    ssh_authorized_key = "${var.ssh_authorized_key}"
+    domain_name           = "${element(var.worker_domains, count.index)}"
+    k8s_dns_service_ip    = "${module.bootkube.kube_dns_service_ip}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"

    # Terraform evaluates both sides regardless and element cannot be used on 0 length lists
    networkd_content = "${length(var.worker_networkds) == 0 ? "" : element(concat(var.worker_networkds, list("")), count.index)}"
--- a/bare-metal/container-linux/kubernetes/variables.tf
+++ b/bare-metal/container-linux/kubernetes/variables.tf
@ -83,7 +83,7 @@ variable "pod_cidr" {
 variable "service_cidr" {
  description = <<EOD
 CIDR IP range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns, the 15th IP will be reserved for self-hosted etcd, and the 200th IP will be reserved for bootstrap self-hosted etcd.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
 EOD

  type    = "string"
@ -92,6 +92,12 @@ EOD

 # optional

+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
+
 variable "cached_install" {
  type        = "string"
  default     = "false"
--- a/bare-metal/container-linux/pxe-worker/cl/bootkube-worker.yaml.tmpl
+++ b/bare-metal/container-linux/pxe-worker/cl/bootkube-worker.yaml.tmpl
@ -50,6 +50,7 @@ systemd:
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
        ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
        ExecStart=/usr/lib/coreos/kubelet-wrapper \
@ -57,7 +58,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns={{.k8s_dns_service_ip}} \
-          --cluster_domain=cluster.local \
+          --cluster_domain={{.cluster_domain_suffix}} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override={{.domain_name}} \
@ -66,7 +67,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --require-kubeconfig
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=5
@ -97,7 +98,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/pxe-worker/groups.tf
+++ b/bare-metal/container-linux/pxe-worker/groups.tf
@ -13,9 +13,10 @@ resource "matchbox_group" "workers" {
    etcd_endpoints = "${join(",", formatlist("%s:2379", var.controller_domains))}"

    # TODO
-    etcd_on_host        = "true"
-    k8s_etcd_service_ip = "10.3.0.15"
-    k8s_dns_service_ip  = "${var.kube_dns_service_ip}"
-    ssh_authorized_key  = "${var.ssh_authorized_key}"
+    etcd_on_host          = "true"
+    k8s_etcd_service_ip   = "10.3.0.15"
+    k8s_dns_service_ip    = "${var.kube_dns_service_ip}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
  }
 }
--- a/bare-metal/container-linux/pxe-worker/profiles.tf
+++ b/bare-metal/container-linux/pxe-worker/profiles.tf
@ -8,6 +8,7 @@ resource "matchbox_profile" "bootkube-worker-pxe" {
  ]

  args = [
+    "initrd=coreos_production_pxe_image.cpio.gz",
    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.first_boot=yes",
    "console=tty0",
--- a/bare-metal/container-linux/pxe-worker/variables.tf
+++ b/bare-metal/container-linux/pxe-worker/variables.tf
@ -64,3 +64,9 @@ variable "kernel_args" {
    "root=/dev/sda1",
  ]
 }
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -1,4 +1,4 @@
-# Typhoon
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -9,9 +9,9 @@ Typhoon is a minimal and free Kubernetes distribution.

 Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.

-## Features
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/digital-ocean/container-linux/kubernetes/bootkube.tf
+++ b/digital-ocean/container-linux/kubernetes/bootkube.tf
@ -1,13 +1,14 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.8.1"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"

-  cluster_name = "${var.cluster_name}"
-  api_servers  = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers = "${digitalocean_record.etcds.*.fqdn}"
-  asset_dir    = "${var.asset_dir}"
-  networking   = "${var.networking}"
-  network_mtu  = 1440
-  pod_cidr     = "${var.pod_cidr}"
-  service_cidr = "${var.service_cidr}"
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  etcd_servers          = "${digitalocean_record.etcds.*.fqdn}"
+  asset_dir             = "${var.asset_dir}"
+  networking            = "${var.networking}"
+  network_mtu           = 1440
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
 }
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -50,6 +50,7 @@ systemd:
        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
        [Install]
        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
    - name: kubelet.service
      contents: |
        [Unit]
@ -83,7 +84,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
@ -92,8 +93,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/master \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-          --require-kubeconfig
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=10
@ -120,7 +120,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -139,11 +139,9 @@ storage:
          # Wrapper for bootkube start
          set -e
          # Move experimental manifests
-          [ -d /opt/bootkube/assets/manifests-* ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          [ -d /opt/bootkube/assets/experimental/manifests ] && mv /opt/bootkube/assets/experimental/manifests/* /opt/bootkube/assets/manifests && rm -r /opt/bootkube/assets/experimental/manifests
-          [ -d /opt/bootkube/assets/experimental/bootstrap-manifests ] && mv /opt/bootkube/assets/experimental/bootstrap-manifests/* /opt/bootkube/assets/bootstrap-manifests && rm -r /opt/bootkube/assets/experimental/bootstrap-manifests
+          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.8.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -60,7 +60,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
@ -68,8 +68,7 @@ systemd:
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --require-kubeconfig
+          --pod-manifest-path=/etc/kubernetes/manifests
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=5
@ -95,7 +94,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -113,7 +112,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.8.2 \
+            docker://gcr.io/google_containers/hyperkube:v1.8.6 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/controllers.tf
+++ b/digital-ocean/container-linux/kubernetes/controllers.tf
@ -14,6 +14,7 @@ resource "digitalocean_record" "controllers" {
  value = "${element(digitalocean_droplet.controllers.*.ipv4_address, count.index)}"
 }

+# Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "digitalocean_record" "etcds" {
  count = "${var.controller_count}"

@ -25,7 +26,7 @@ resource "digitalocean_record" "etcds" {
  type = "A"
  ttl  = 300

-  # IPv4 addresses of controllers
+  # private IPv4 address for etcd
  value = "${element(digitalocean_droplet.controllers.*.ipv4_address_private, count.index)}"
 }

@ -68,8 +69,9 @@ data "template_file" "controller_config" {
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"

    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", formatlist("%s=https://%s:2380", null_resource.repeat.*.triggers.name, null_resource.repeat.*.triggers.domain))}"
-    k8s_dns_service_ip   = "${cidrhost(var.service_cidr, 10)}"
+    etcd_initial_cluster  = "${join(",", formatlist("%s=https://%s:2380", null_resource.repeat.*.triggers.name, null_resource.repeat.*.triggers.domain))}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
  }
 }

--- a/digital-ocean/container-linux/kubernetes/ssh.tf
+++ b/digital-ocean/container-linux/kubernetes/ssh.tf
@ -69,7 +69,7 @@ resource "null_resource" "copy-secrets" {
 # Secure copy bootkube assets to ONE controller and start bootkube to perform
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
-  depends_on = ["module.bootkube", "null_resource.copy-secrets", "digitalocean_droplet.controllers"]
+  depends_on = ["module.bootkube", "null_resource.copy-secrets"]

  connection {
    type    = "ssh"
--- a/digital-ocean/container-linux/kubernetes/variables.tf
+++ b/digital-ocean/container-linux/kubernetes/variables.tf
@ -70,9 +70,16 @@ variable "pod_cidr" {
 variable "service_cidr" {
  description = <<EOD
 CIDR IP range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns, the 15th IP will be reserved for self-hosted etcd, and the 200th IP will be reserved for bootstrap self-hosted etcd.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
 EOD

  type    = "string"
  default = "10.3.0.0/16"
 }
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
+
--- a/digital-ocean/container-linux/kubernetes/workers.tf
+++ b/digital-ocean/container-linux/kubernetes/workers.tf
@ -43,8 +43,9 @@ data "template_file" "worker_config" {
  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"

  vars = {
-    k8s_dns_service_ip  = "${cidrhost(var.service_cidr, 10)}"
-    k8s_etcd_service_ip = "${cidrhost(var.service_cidr, 15)}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    k8s_etcd_service_ip   = "${cidrhost(var.service_cidr, 15)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
  }
 }

--- a/docs/addons/cluo.md
+++ b/docs/addons/cluo.md
@ -12,13 +12,13 @@ kubectl apply -f addons/cluo -R

 ## Usage

-`update-agent` runs as a DaemonSet and annotates a node when `update-engine.service` indiates an update has been installed and a reboot is needed. It also adds additional labels and annotations to nodes.
+`update-agent` runs as a DaemonSet and annotates a node when `update-engine.service` indicates an update has been installed and a reboot is needed. It also adds additional labels and annotations to nodes.

 ```
 $ kubectl get nodes --show-labels
 ...
 container-linux-update.v1.coreos.com/group=stable
-container-linux-update.v1.coreos.com/version=1465.6.0
+container-linux-update.v1.coreos.com/version=1576.4.0
 ```

 `update-operator` ensures one node reboots at a time and that pods are drained prior to reboot.
--- a/docs/addons/dashboard.md
+++ b/docs/addons/dashboard.md
@ -1,5 +1,8 @@
 # Kubernetes Dashboard

+!!! warning
+    The Kubernetes Dashboard takes [unusual approaches](https://github.com/kubernetes/dashboard/wiki/Access-control#authorization-header) to security and is often a point of security escalations. We recommend you do don't deploy it and get familiar with `kubectl`, if possible.
+
 The Kubernetes [Dashboard](https://github.com/kubernetes/dashboard) provides a web UI to manage a Kubernetes cluster for those who prefer an alternative to `kubectl`.

 ## Create
--- a/docs/addons/heapster.md
+++ b/docs/addons/heapster.md
@ -1,6 +1,6 @@
 # Heapster

-[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command and Kubernetes dashbard graphs.
+[Heapster](https://kubernetes.io/docs/user-guide/monitoring/) collects data from apiservers and kubelets and exposes it through a REST API. This API powers the `kubectl top` command and Kubernetes dashboard graphs.

 ## Create

--- a/docs/addons/overview.md
+++ b/docs/addons/overview.md
@ -5,7 +5,7 @@ Every Typhoon cluster is verified to work well with several post-install addons.
 * [CLUO](cluo.md) (Container Linux only)
 * Nginx [Ingress Controller](ingress.md)
 * [Heapster](heapster.md)
-* Kubernetes [Dashboard](dashboard.md)
 * [Prometheus](prometheus.md)
-* Grafana
+* [Grafana](prometheus.md#grafana)
+* Kubernetes [Dashboard](dashboard.md)

--- a/docs/addons/prometheus.md
+++ b/docs/addons/prometheus.md
@ -30,9 +30,9 @@ Exporters expose metrics for 3rd-party systems that don't natively expose Promet
 * [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) - Deployment that exposes Kubernetes object metrics
 * [blackbox_exporter](https://github.com/prometheus/blackbox_exporter) - Scrapes HTTP, HTTPS, DNS, TCP, or ICMP endpoints and exposes availability as metrics

-### Queries and Graphs
+### Queries and Alerts

-Prometheus provides a simplistic UI for querying and graphing metrics. Use `kubectl` to authenticate to the apiserver and create a local port-forward to the Prometheus pod.
+Prometheus provides a simplistic UI for querying metrics and viewing alerts. Use `kubectl` to authenticate to the apiserver and create a local port-forward to the Prometheus pod.

 ```
 kubectl get pods -n monitoring
@ -44,7 +44,24 @@ Visit [127.0.0.1:9090](http://127.0.0.1:9090) to query [expressions](http://127.
 ![Prometheus Graph](/img/prometheus-graph.png)
 <br/>
 ![Prometheus Targets](/img/prometheus-targets.png)
+<br/>
+![Prometheus Alerts](/img/prometheus-alerts.png)

-### Visualization
+## Grafana
+
+Grafana can be used to build dashboards and rich visualizations that use Prometheus as the datasource. Create the grafana deployment and service.
+
+```
+kubectl apply -f addons/grafana -R
+```
+
+Use `kubectl` to authenticate to the apiserver and create a local port-forward to the Grafana pod.
+
+```
+kubectl port-forward grafana-POD-ID 8080 -n monitoring
+```
+
+Visit [127.0.0.1:8080](http://127.0.0.1:8080), add the prometheus data-source (http://prometheus.monitoring.svc.cluster.local), and import your desired dashboard (e.g. [Grafana Dashboard 315](https://grafana.com/dashboards/315)).
+
+![Grafana Dashboard](/img/grafana-dashboard.png)

-Grafana can be used to build dashboards and rich visualizations that use Prometheus as the datasource. Favor Grafana for these use cases and use the Prometheus for debugging or quickly checking available metrics.
--- a/docs/aws.md
+++ b/docs/aws.md
@ -1,26 +1,20 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.8.2 cluster on AWS.
+In this tutorial, we'll create a Kubernetes v1.8.6 cluster on AWS.

 We'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module. On apply, a VPC, gateway, subnets, auto-scaling groups of controllers and workers, network load balancers for controllers and workers, and security groups will be created.

-Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules `etcd`, `apiserver`, `scheduler`, `controller-manager`, and `kube-dns` on controllers and runs `kube-proxy` and `flannel` or `calico` on each node. A generated `kubeconfig` provides `kubectl` access to the cluster.
-
-!!! warning "Alpha"
-    Typhoon Kubernetes clusters on AWS are marked as "alpha".
-
-!!! warning "Disabled"
-    Clusters do not use EC2 instances with elevated IAM roles. Kubernetes AWS integrations are not enabled.
+Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules an `apiserver`, `scheduler`, `controller-manager`, and `kube-dns` on controllers and runs `kube-proxy` and `calico` or `flannel` on each node. A generated `kubeconfig` provides `kubectl` access to the cluster.

 ## Requirements

 * AWS Account and IAM credentials
 * AWS Route53 DNS Zone (registered Domain Name or delegated subdomain)
-* Terraform v0.10.4+ and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
+* Terraform v0.10.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.10.1 on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.

 ```sh
 $ terraform version
@ -87,7 +81,7 @@ module "aws-tempest" {
  dns_zone           = "aws.example.com"
  dns_zone_id        = "Z3PAABBCFAKEC0"
  controller_count   = 1
-  controller_type    = "t2.small"
+  controller_type    = "t2.medium"
  worker_count       = 2
  worker_type        = "t2.small"
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
@ -109,7 +103,7 @@ ssh-add -L
 ```

 !!! warning
-    `terrafrom apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.
+    `terraform apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.

 ## Apply

@ -125,7 +119,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.8.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
 ```

 Plan the resources to be created.
@ -140,17 +134,14 @@ Apply the changes to create the cluster.
 ```sh
 $ terraform apply
 ...
-module.aws-tempest.null_resource.bootkube-start: Still creating... (10m50s elapsed)
-module.aws-tempest.null_resource.bootkube-start: Still creating... (11m0s elapsed)
+module.aws-tempest.null_resource.bootkube-start: Still creating... (4m50s elapsed)
+module.aws-tempest.null_resource.bootkube-start: Still creating... (5m0s elapsed)
 module.aws-tempest.null_resource.bootkube-start: Creation complete after 11m8s (ID: 3961816482286168143)

 Apply complete! Resources: 98 added, 0 changed, 0 destroyed.
 ```

-In 10-20 minutes, the Kubernetes cluster will be ready.
-
-!!! bug ""
-    Typhoon clusters on AWS take much longer to create than clusters on other platforms. This is related to DNS resolution time to the ASG, which will be resolved in a future version that uses static controllers.
+In 4-8 minutes, the Kubernetes cluster will be ready.

 ## Verify

@ -160,9 +151,9 @@ In 10-20 minutes, the Kubernetes cluster will be ready.
 $ KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.8.2
-ip-10-0-19-112   Ready     34m       v1.8.2
-ip-10-0-4-22     Ready     34m       v1.8.2
+ip-10-0-12-221   Ready     34m       v1.8.6
+ip-10-0-19-112   Ready     34m       v1.8.6
+ip-10-0-4-22     Ready     34m       v1.8.6
 ```

 List the pods.
@ -173,13 +164,10 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1m5bf                         2/2    Running   0         34m              
 kube-system   calico-node-7jmr1                         2/2    Running   0         34m              
 kube-system   calico-node-bknc8                         2/2    Running   0         34m              
-kube-system   etcd-operator-2287495111-br512            1/1    Running   1         34m              
 kube-system   kube-apiserver-4mjbk                      1/1    Running   0         34m              
 kube-system   kube-controller-manager-3597210155-j2jbt  1/1    Running   1         34m              
 kube-system   kube-controller-manager-3597210155-j7g7x  1/1    Running   0         34m              
 kube-system   kube-dns-1187388186-wx1lg                 3/3    Running   0         34m              
-kube-system   kube-etcd-0000                            1/1    Running   0         32m              
-kube-system   kube-etcd-network-checkpointer-dt5pt      1/1    Running   0         34m              
 kube-system   kube-proxy-14wxv                          1/1    Running   0         34m              
 kube-system   kube-proxy-9vxh2                          1/1    Running   0         34m              
 kube-system   kube-proxy-sbbsh                          1/1    Running   0         34m              
@ -213,7 +201,7 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]

 Clusters create a DNS A record `${cluster_name}.${dns_zone}` to resolve a network load balancer backed by controller instances. This FQDN is used by workers and `kubectl` to access the apiserver. In this example, the cluster's apiserver would be accessible at `tempest.aws.example.com`.

-You'll need a registered domain name or subdomain registered in a AWS Route53 DNS zone. You can set this up once and create many clusters with unqiue names.
+You'll need a registered domain name or subdomain registered in a AWS Route53 DNS zone. You can set this up once and create many clusters with unique names.

 ```tf
 resource "aws_route53_zone" "zone-for-clusters" {
@ -239,7 +227,8 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
 | host_cidr | CIDR range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |
 | pod_cidr | CIDR range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
-| service_cidr | CIDR range to assgin to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| service_cidr | CIDR range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by kube-dns. | "cluster.local" | "k8s.example.com" |

 Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/).

--- a/docs/bare-metal.md
+++ b/docs/bare-metal.md
@ -1,10 +1,10 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provison a Kubernetes v1.8.2 cluster on bare-metal.
+In this tutorial, we'll network boot and provision a Kubernetes v1.8.6 cluster on bare-metal.

 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers.

-Controllers are provisioned as etcd peers and run `etcd-member` (etcd3) and `kubelet`. Workers are provisioned to run a `kubelet`. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules an `apiserver`, `scheduler`, `controller-manager`, and `kube-dns` on controllers and runs `kube-proxy` and `flannel` or `calico` on each node. A generated `kubeconfig` provides `kubectl` access to the cluster.
+Controllers are provisioned as etcd peers and run `etcd-member` (etcd3) and `kubelet`. Workers are provisioned to run a `kubelet`. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules an `apiserver`, `scheduler`, `controller-manager`, and `kube-dns` on controllers and runs `kube-proxy` and `calico` or `flannel` on each node. A generated `kubeconfig` provides `kubectl` access to the cluster.

 ## Requirements

@ -12,7 +12,7 @@ Controllers are provisioned as etcd peers and run `etcd-member` (etcd3) and `kub
 * PXE-enabled [network boot](https://coreos.com/matchbox/docs/latest/network-setup.html) environment
 * Matchbox v0.6+ deployment with API enabled
 * Matchbox credentials `client.crt`, `client.key`, `ca.crt`
-* Terraform v0.10.4+ and [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) installed locally
+* Terraform v0.10.x and [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) installed locally

 ## Machines

@ -31,7 +31,7 @@ Configure each machine to boot from the disk [^1] through IPMI or the BIOS menu.
 ipmitool -H node1 -U USER -P PASS chassis bootdev disk options=persistent
 ```
 
-During provisioning, you'll explicitly set the boot device to `pxe` for the next boot only. Machines will install (overwrite) the operting system to disk on PXE boot and reboot into the disk install.
+During provisioning, you'll explicitly set the boot device to `pxe` for the next boot only. Machines will install (overwrite) the operating system to disk on PXE boot and reboot into the disk install.

 !!! tip ""
    Ask your hardware vendor to provide MACs and preconfigure IPMI, if possible. With it, you can rack new servers, `terraform apply` with new info, and power on machines that network boot and provision into clusters.
@ -105,11 +105,11 @@ Read about the [many ways](https://coreos.com/matchbox/docs/latest/network-setup
 * Place Matchbox behind a menu entry (timeout and default to Matchbox)

 !!! note ""
-    TFTP chainloding to modern boot firmware, like iPXE, avoids issues with old NICs and allows faster transfer protocols like HTTP to be used.
+    TFTP chainloading to modern boot firmware, like iPXE, avoids issues with old NICs and allows faster transfer protocols like HTTP to be used.

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.9.2+ on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.

 ```sh
 $ terraform version
@ -162,7 +162,7 @@ module "bare-metal-mercury" {
  # install
  matchbox_http_endpoint  = "http://matchbox.example.com"
  container_linux_channel = "stable"
-  container_linux_version = "1520.6.0"
+  container_linux_version = "1576.4.0"
  ssh_authorized_key      = "ssh-rsa AAAAB3Nz..."

  # cluster
@ -203,7 +203,7 @@ ssh-add -L
 ```

 !!! warning
-    `terrafrom apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.
+    `terraform apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.

 ## Apply

@ -219,7 +219,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.8.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
 ```

 Plan the resources to be created.
@ -290,9 +290,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.8.2
-node2.example.com   Ready     11m       v1.8.2
-node3.example.com   Ready     11m       v1.8.2
+node1.example.com   Ready     11m       v1.8.6
+node2.example.com   Ready     11m       v1.8.6
+node3.example.com   Ready     11m       v1.8.6
 ```

 List the pods.
@ -332,7 +332,7 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]
 |:-----|:------------|:--------|
 | matchbox_http_endpoint | Matchbox HTTP read-only endpoint | http://matchbox.example.com:8080 |
 | container_linux_channel | Container Linux channel | stable, beta, alpha |
-| container_linux_version | Container Linux version of the kernel/initrd to PXE and the image to install | 1465.6.0 |
+| container_linux_version | Container Linux version of the kernel/initrd to PXE and the image to install | 1576.4.0 |
 | cluster_name | Cluster name | mercury |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
 | ssh_authorized_key | SSH public key for ~/.ssh/authorized_keys | "ssh-rsa AAAAB3Nz..." |
@ -354,6 +354,7 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico-only) | 1480 | - | 
 | pod_cidr | CIDR range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
-| service_cidr | CIDR range to assgin to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| service_cidr | CIDR range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by kube-dns. | "cluster.local" | "k8s.example.com" |
 | kernel_args | Additional kernel args to provide at PXE boot | [] | "kvm-intel.nested=1" |

--- a/docs/digital-ocean.md
+++ b/docs/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.8.2 cluster on Digital Ocean.
+In this tutorial, we'll create a Kubernetes v1.8.6 cluster on Digital Ocean.

 We'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module. On apply, firewall rules, DNS records, tags, and droplets for Kubernetes controllers and workers will be created.

@ -10,11 +10,11 @@ Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube

 * Digital Ocean Account and Token
 * Digital Ocean Domain (registered Domain Name or delegated subdomain)
-* Terraform v0.10.4+ and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
+* Terraform v0.10.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.10.1+ on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.

 ```sh
 $ terraform version
@ -98,7 +98,7 @@ ssh-add -L
 ```

 !!! warning
-    `terrafrom apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.
+    `terraform apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.

 ## Apply

@ -114,7 +114,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.8.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
 ```

 Plan the resources to be created.
@ -147,9 +147,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.8.2
-10.132.115.81    Ready     10m       v1.8.2
-10.132.124.107   Ready     10m       v1.8.2
+10.132.110.130   Ready     10m       v1.8.6
+10.132.115.81    Ready     10m       v1.8.6
+10.132.124.107   Ready     10m       v1.8.6
 ```

 List the pods.
@ -195,7 +195,7 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]

 Clusters create DNS A records `${cluster_name}.${dns_zone}` to resolve to controller droplets (round robin). This FQDN is used by workers and `kubectl` to access the apiserver. In this example, the cluster's apiserver would be accessible at `nemo.do.example.com`.

-You'll need a registered domain name or subdomain registered in Digital Ocean Domains (i.e. DNS zones). You can set this up once and create many clusters with unqiue names.
+You'll need a registered domain name or subdomain registered in Digital Ocean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.

 ```tf
 resource "digitalocean_domain" "zone-for-clusters" {
@ -237,7 +237,8 @@ If you uploaded an SSH key to DigitalOcean (not required), find the fingerprint
 | worker_type | Digital Ocean droplet size | 512mb | 512mb, 1gb, 2gb, 4gb |
 | networking | Choice of networking provider | "flannel" | "flannel" |
 | pod_cidr | CIDR range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
-| service_cidr | CIDR range to assgin to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| service_cidr | CIDR range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by kube-dns. | "cluster.local" | "k8s.example.com" |

 !!! warning
    Do not choose a `controller_type` smaller than `2gb`. The `1gb` droplet is not sufficient for running a controller and bootstrapping will fail.
--- a/docs/faq.md
+++ b/docs/faq.md
@ -6,15 +6,6 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 Formats rise and evolve. Typhoon may choose to adapt the format over time (with lots of forewarning). However, the authors' have built several Kubernetes "distros" before and learned from mistakes - Terraform modules are the right format for now.

-## Self-hosted etcd
-
-AWS and Google Cloud clusters run etcd as "self-hosted" pods, managed by the [etcd-operator](https://github.com/coreos/etcd-operator). By contrast, Typhoon bare-metal and Digital Ocean run an etcd peer as a systemd `etcd-member.service` on each controller (i.e. on-host).
-
-In practice, self-hosted etcd has proven to be *ok*, but not ideal. Running the apiserver's etcd atop Kubernetes itself is inherently complex, but works in most cases. It can be opaque to debug if complex edge cases with upstream Kubernetes bugs arise.
-
-!!! note ""
-    Over time, we plan to deprecate self-hosted etcd and revert to running etcd on-host.
-
 ## Operating Systems

 Only Container Linux is supported currently. This just due to operational familiarity, rather than intentional exclusion. It's important that another operating system be added, to reduce the risk of making narrowly-scoped design decisions.
@ -27,7 +18,7 @@ Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/

 ## Security Issues

-If you find security issues, please see [security disclosures](/topics/security).
+If you find security issues, please see [security disclosures](/topics/security.md#disclosures).

 ## Maintainers

--- a/docs/google-cloud.md
+++ b/docs/google-cloud.md
@ -1,20 +1,20 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.8.2 cluster on Google Compute Engine (not GKE).
+In this tutorial, we'll create a Kubernetes v1.8.6 cluster on Google Compute Engine (not GKE).

 We'll declare a Kubernetes cluster in Terraform using the Typhoon Terraform module. On apply, a network, firewall rules, managed instance groups of Kubernetes controllers and workers, network load balancers for controllers and workers, and health checks will be created.

-Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules `etcd`, `apiserver`, `scheduler`, `controller-manager`, and `kube-dns` on controllers and runs `kube-proxy` and `flannel` on each node. A generated `kubeconfig` provides `kubectl` access to the cluster.
+Controllers and workers are provisioned to run a `kubelet`. A one-time [bootkube](https://github.com/kubernetes-incubator/bootkube) bootstrap schedules an `apiserver`, `scheduler`, `controller-manager`, and `kube-dns` on controllers and runs `kube-proxy` and `calico` or `flannel` on each node. A generated `kubeconfig` provides `kubectl` access to the cluster.

 ## Requirements

 * Google Cloud Account and Service Account
 * Google Cloud DNS Zone (registered Domain Name or delegated subdomain)
-* Terraform v0.10.4+ and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally
+* Terraform v0.10.x and [terraform-provider-ct](https://github.com/coreos/terraform-provider-ct) installed locally

 ## Terraform Setup

-Install [Terraform](https://www.terraform.io/downloads.html) v0.9.2+ on your system.
+Install [Terraform](https://www.terraform.io/downloads.html) v0.10.x on your system.

 ```sh
 $ terraform version
@ -77,10 +77,10 @@ module "google-cloud-yavin" {
  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"

  # Google Cloud
-  zone          = "us-central1-c"
+  region        = "us-central1"
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"
-  os_image      = "coreos-stable-1520-6-0-v20171012"
+  os_image      = "coreos-stable-1576-4-0-v20171206"

  cluster_name       = "yavin"
  controller_count   = 1
@ -104,7 +104,7 @@ ssh-add -L
 ```

 !!! warning
-    `terrafrom apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.
+    `terraform apply` will hang connecting to a controller if `ssh-agent` does not contain the SSH key.

 ## Apply

@ -120,7 +120,7 @@ Get or update Terraform modules.
 $ terraform get            # downloads missing modules
 $ terraform get --update   # updates all modules
 Get: git::https://github.com/poseidon/typhoon (update)
-Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.8.1 (update)
+Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.9.1 (update)
 ```

 Plan the resources to be created.
@ -137,14 +137,14 @@ $ terraform apply
 module.google-cloud-yavin.null_resource.bootkube-start: Still creating... (10s elapsed)
 ...

-module.google-cloud-yavin.null_resource.bootkube-start: Still creating... (8m30s elapsed)
-module.google-cloud-yavin.null_resource.bootkube-start: Still creating... (8m40s elapsed)
+module.google-cloud-yavin.null_resource.bootkube-start: Still creating... (5m30s elapsed)
+module.google-cloud-yavin.null_resource.bootkube-start: Still creating... (5m40s elapsed)
 module.google-cloud-yavin.null_resource.bootkube-start: Creation complete (ID: 5768638456220583358)

 Apply complete! Resources: 64 added, 0 changed, 0 destroyed.
 ```

-In 5-10 minutes, the Kubernetes cluster will be ready.
+In 4-8 minutes, the Kubernetes cluster will be ready.

 ## Verify

@ -154,9 +154,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-1682.c.example-com.internal  Ready    6m     v1.8.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.8.6
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.6
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.6
 ```

 List the pods.
@ -167,13 +167,10 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
-kube-system   etcd-operator-3329263108-f443m            1/1    Running   1         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
 kube-system   kube-dns-1187388186-zj5dl                 3/3    Running   0         6m
-kube-system   kube-etcd-0000                            1/1    Running   0         5m
-kube-system   kube-etcd-network-checkpointer-crznb      1/1    Running   0         6m
 kube-system   kube-proxy-117v6                          1/1    Running   0         6m
 kube-system   kube-proxy-9886n                          1/1    Running   0         6m
 kube-system   kube-proxy-njn47                          1/1    Running   0         6m
@ -196,20 +193,20 @@ Learn about [version pinning](concepts.md#versioning), maintenance, and [addons]
 | Name | Description | Example |
 |:-----|:------------|:--------|
 | cluster_name | Unique cluster name (prepended to dns_zone) | "yavin" |
-| zone | Google Cloud zone | "us-central1-f" |
+| region | Google Cloud region | "us-central1" |
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
 | ssh_authorized_key | SSH public key for ~/.ssh_authorized_keys | "ssh-rsa AAAAB3NZ..." |
-| os_image | OS image for compute instances | "coreos-stable-1465-6-0-v20170817" |
+| os_image | OS image for compute instances | "coreos-stable-1576-4-0-v20171206" |
 | asset_dir | Path to a directory where generated assets should be placed (contains secrets) | "/home/user/.secrets/clusters/yavin" |

-Check the list of valid [zones](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Container Linux [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep coreos`.
+Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Container Linux [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep coreos`.

 #### DNS Zone

 Clusters create a DNS A record `${cluster_name}.${dns_zone}` to resolve a network load balancer backed by controller instances. This FQDN is used by workers and `kubectl` to access the apiserver. In this example, the cluster's apiserver would be accessible at `yavin.google-cloud.example.com`.

-You'll need a registered domain name or subdomain registered in a Google Cloud DNS zone. You can set this up once and create many clusters with unqiue names.
+You'll need a registered domain name or subdomain registered in a Google Cloud DNS zone. You can set this up once and create many clusters with unique names.

 ```tf
 resource "google_dns_managed_zone" "zone-for-clusters" {
@ -232,11 +229,15 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | worker_preemptible | If enabled, Compute Engine will terminate controllers randomly within 24 hours | false | true |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | pod_cidr | CIDR range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
-| service_cidr | CIDR range to assgin to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| service_cidr | CIDR range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| cluster_domain_suffix | FQDN suffix for Kubernetes services answered by kube-dns. | "cluster.local" | "k8s.example.com" |

 Check the list of valid [machine types](https://cloud.google.com/compute/docs/machine-types).

+!!! warning
+    Set controller_count to 1. A bug in Google Cloud network load balancer health checking prevents multiple controllers from bootstrapping. There are workarounds, but they all involve tradeoffs we're uncomfortable recommending. See [#54](https://github.com/poseidon/typhoon/issues/54).
+
 #### Preemption

-Add `worker_premeptible = "true"` to allow worker nodes to be [preempted](https://cloud.google.com/compute/docs/instances/preemptible) at random, but pay [significantly](https://cloud.google.com/compute/pricing) less. Clusters tolerate stopping instances fairly well (reschedules pods, but cannot drain) and preemption provides a nice reward for running fault-tolerant cluster systems.`
+Add `worker_preemeptible = "true"` to allow worker nodes to be [preempted](https://cloud.google.com/compute/docs/instances/preemptible) at random, but pay [significantly](https://cloud.google.com/compute/pricing) less. Clusters tolerate stopping instances fairly well (reschedules pods, but cannot drain) and preemption provides a nice reward for running fault-tolerant cluster systems.`

--- a/docs/img/favicon.ico
+++ b/docs/img/favicon.ico
--- a/docs/img/grafana-dashboard.png
+++ b/docs/img/grafana-dashboard.png
--- a/docs/img/prometheus-alerts.png
+++ b/docs/img/prometheus-alerts.png
--- a/docs/img/spin.png
+++ b/docs/img/spin.png
--- a/docs/img/typhoon-logo.png
+++ b/docs/img/typhoon-logo.png
--- a/docs/img/typhoon.png
+++ b/docs/img/typhoon.png
--- a/docs/index.md
+++ b/docs/index.md
@ -1,4 +1,4 @@
-# Typhoon <img align="right" src="https://storage.googleapis.com/dghubble/spin.png">
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -9,9 +9,9 @@ Typhoon is a minimal and free Kubernetes distribution.

 Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.

-## Features
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics and other optional [addons](addons/overview.md)
@ -23,7 +23,7 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws.md) | alpha |
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws.md) | beta |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal.md) | stable |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean.md) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud.md) | beta |
@ -46,10 +46,10 @@ module "google-cloud-yavin" {
  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes"

  # Google Cloud
-  zone          = "us-central1-c"
+  region        = "us-central1"
  dns_zone      = "example.com"
  dns_zone_name = "example-zone"
-  os_image      = "coreos-stable-1465-6-0-v20170817"
+  os_image      = "coreos-stable-1576-4-0-v20171206"

  cluster_name       = "yavin"
  controller_count   = 1
@ -71,15 +71,15 @@ $ terraform apply
 Apply complete! Resources: 64 added, 0 changed, 0 destroyed.
 ```

-In 5-10 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.
+In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.

 ```
 $ KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-1682.c.example-com.internal  Ready    6m     v1.8.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.8.6
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.8.6
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.8.6
 ```

 List the pods.
@ -90,13 +90,10 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
-kube-system   etcd-operator-3329263108-f443m            1/1    Running   1         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
 kube-system   kube-dns-1187388186-zj5dl                 3/3    Running   0         6m
-kube-system   kube-etcd-0000                            1/1    Running   0         5m
-kube-system   kube-etcd-network-checkpointer-crznb      1/1    Running   0         6m
 kube-system   kube-proxy-117v6                          1/1    Running   0         6m
 kube-system   kube-proxy-9886n                          1/1    Running   0         6m
 kube-system   kube-proxy-njn47                          1/1    Running   0         6m
--- a/docs/topics/hardware.md
+++ b/docs/topics/hardware.md
@ -2,9 +2,9 @@

 While bare-metal Kubernetes clusters have no special hardware requirements (beyond the [min reqs](/bare-metal.md#requirements)), Typhoon does ensure certain router and server hardware integrates well with Kubernetes.

-## Ubiquitiy
+## Ubiquiti

-Ubiquity EdgeRouters work well with bare-metal Kubernetes clusters. Knowledge about how to setup an EdgeRouter and use the CLI is required.
+Ubiquiti EdgeRouters work well with bare-metal Kubernetes clusters. Knowledge about how to setup an EdgeRouter and use the CLI is required.

 ### PXE

@ -50,7 +50,7 @@ Add `dnsmasq` command line options to enable the TFTP file server.

 ```
 configure
-show servce dns forwarding
+show service dns forwarding
 set service dns forwarding options enable-tftp
 set service dns forwarding options tftp-root=/var/lib/tftpboot
 commit-confirm
@ -143,7 +143,7 @@ commit-confirm

 ### BGP

-Add the EdgeRouter as a global BGP peer for nodes in a Kubernetes cluster (requires Calico). Neighbors will exchange `podCIDR` routes and individual pods will become routeable on the LAN.
+Add the EdgeRouter as a global BGP peer for nodes in a Kubernetes cluster (requires Calico). Neighbors will exchange `podCIDR` routes and individual pods will become routable on the LAN.

 Configure node(s) as BGP neighbors.

--- a/docs/topics/performance.md
+++ b/docs/topics/performance.md
@ -6,21 +6,20 @@ Provisioning times vary based on the platform. Sampling the time to create (appl

 | Platform      | Apply | Destroy |
 |---------------|-------|---------|
-| AWS           | 20 min | 8 min 10 sec |
+| AWS           | 6 min | 5 min   |
 | Bare-Metal    | 10-14 min | NA  |
 | Digital Ocean | 3 min 30 sec | 20 sec |
-| Google Cloud  | 6 min 10 sec | 4 min 30 sec |
+| Google Cloud  | 4 min | 4 min 30 sec |

 Notes:

-* AWS is alpha
-* DNS propagation times have a large impact on provision time
+* SOA TTL and NXDOMAIN caching can have a large impact on provision time
 * Platforms with auto-scaling take more time to provision (AWS, Google)
-* Bare-metal provision times vary depending on the time for machines to POST and network bandwidth to download images.
+* Bare-metal POST times and network bandwidth will affect provision times

 ## Network Performance

-Network performance varies based on the platform and CNI plugin. `iperf` was used to measture the bandwidth between different hosts and different pods. Host-to-host indicates the typical bandwidth offered by the provider. Pod-to-pod shows the bandwidth between two `iperf` containers. The difference provides some idea about the overhead.
+Network performance varies based on the platform and CNI plugin. `iperf` was used to measure the bandwidth between different hosts and different pods. Host-to-host shows typical bandwidth between host machines. Pod-to-pod shows the bandwidth between two `iperf` containers.

 | Platform / Plugin          | Theory | Host to Host | Pod to Pod   |
 |----------------------------|-------:|-------------:|-------------:|
@ -37,10 +36,7 @@ Network performance varies based on the platform and CNI plugin. `iperf` was use

 Notes:

-* AWS is alpha
-* AWS instances are located in the same region. Google instances are located in the same zone (helps bandwidth at the expense of fault tolerance).
-* Network bandwidth fluctuates on AWS and Digital Ocean.
+* Calico and Flannel have comparable performance. Platform and configuration differences dominate.
+* Neither CNI provider seems to be able to leverage bonded NICs (bare-metal)
+* AWS and Digital Ocean network bandwidth fluctuates more than on other platforms.
 * Only [certain AWS EC2 instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) allow jumbo frames. This is why the default MTU on AWS must be 1480.
-* Between Flannel and Calico, performance differences are usually minimal. Platform and configuration differenes dominate.
-* Pods do not seem to be able to leverage the hosts' bonded NIC setup. Possibly a testing artifact.
-* Observing the same bonded NIC pod-to-pod limit suggests the bottleneck lies below flannel and calico.
--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -1,4 +1,4 @@
-# Typhoon
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -9,9 +9,9 @@ Typhoon is a minimal and free Kubernetes distribution.

 Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.

-## Features
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.8.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.8.6 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Dashboards, Metrics, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
--- a/google-cloud/container-linux/kubernetes/bootkube.tf
+++ b/google-cloud/container-linux/kubernetes/bootkube.tf
@ -1,14 +1,14 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=v0.8.1"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5072569bb7dff1c2f6bc6fb7b06ce0a41809971e"

-  cluster_name                  = "${var.cluster_name}"
-  api_servers                   = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers                  = ["http://127.0.0.1:2379"]
-  asset_dir                     = "${var.asset_dir}"
-  networking                    = "${var.networking}"
-  network_mtu                   = 1440
-  pod_cidr                      = "${var.pod_cidr}"
-  service_cidr                  = "${var.service_cidr}"
-  experimental_self_hosted_etcd = "true"
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  etcd_servers          = "${module.controllers.etcd_fqdns}"
+  asset_dir             = "${var.asset_dir}"
+  networking            = "${var.networking}"
+  network_mtu           = 1440
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
 }
--- a/google-cloud/container-linux/kubernetes/cluster.tf
+++ b/google-cloud/container-linux/kubernetes/cluster.tf
@ -6,16 +6,16 @@ module "controllers" {
  # GCE
  network       = "${google_compute_network.network.name}"
  count         = "${var.controller_count}"
-  zone          = "${var.zone}"
+  region        = "${var.region}"
  dns_zone      = "${var.dns_zone}"
  dns_zone_name = "${var.dns_zone_name}"
  machine_type  = "${var.machine_type}"
  os_image      = "${var.os_image}"
-  preemptible   = "${var.controller_preemptible}"

  # configuration
  networking              = "${var.networking}"
  service_cidr            = "${var.service_cidr}"
+  cluster_domain_suffix   = "${var.cluster_domain_suffix}"
  kubeconfig_ca_cert      = "${module.bootkube.ca_cert}"
  kubeconfig_kubelet_cert = "${module.bootkube.kubelet_cert}"
  kubeconfig_kubelet_key  = "${module.bootkube.kubelet_key}"
@ -29,14 +29,15 @@ module "workers" {

  # GCE
  network      = "${google_compute_network.network.name}"
+  region       = "${var.region}"
  count        = "${var.worker_count}"
-  zone         = "${var.zone}"
  machine_type = "${var.machine_type}"
  os_image     = "${var.os_image}"
  preemptible  = "${var.worker_preemptible}"

  # configuration
  service_cidr            = "${var.service_cidr}"
+  cluster_domain_suffix   = "${var.cluster_domain_suffix}"
  kubeconfig_ca_cert      = "${module.bootkube.ca_cert}"
  kubeconfig_kubelet_cert = "${module.bootkube.kubelet_cert}"
  kubeconfig_kubelet_key  = "${module.bootkube.kubelet_key}"
--- a/google-cloud/container-linux/kubernetes/controllers/cl/controller.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/controllers/cl/controller.yaml.tmpl
@ -1,6 +1,29 @@
 ---
 systemd:
  units:
+    - name: etcd-member.service
+      enable: true
+      dropins:
+        - name: 40-etcd-cluster.conf
+          contents: |
+            [Service]
+            Environment="ETCD_IMAGE_TAG=v3.2.0"
+            Environment="ETCD_NAME=${etcd_name}"
+            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
+            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
+            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
+            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
+            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
+            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
+            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
+            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
+            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
+            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
+            Environment="ETCD_CLIENT_CERT_AUTH=true"
+            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
+            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
+            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
+            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
    - name: docker.service
      enable: true
    - name: locksmithd.service
@ -18,6 +41,7 @@ systemd:
        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
        [Install]
        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
    - name: kubelet.service
      enable: true
      contents: |
@ -50,7 +74,7 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --kubeconfig=/etc/kubernetes/kubeconfig \
@ -58,8 +82,7 @@ systemd:
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/master \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-          --require-kubeconfig
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=10
@ -107,7 +130,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -126,11 +149,9 @@ storage:
          # Wrapper for bootkube start
          set -e
          # Move experimental manifests
-          [ -d /opt/bootkube/assets/manifests-* ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
-          [ -d /opt/bootkube/assets/experimental/manifests ] && mv /opt/bootkube/assets/experimental/manifests/* /opt/bootkube/assets/manifests && rm -r /opt/bootkube/assets/experimental/manifests
-          [ -d /opt/bootkube/assets/experimental/bootstrap-manifests ] && mv /opt/bootkube/assets/experimental/bootstrap-manifests/* /opt/bootkube/assets/bootstrap-manifests && rm -r /opt/bootkube/assets/experimental/bootstrap-manifests
+          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.8.1}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.9.1}"
          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
          exec /usr/bin/rkt run \
            --trust-keys-from-https \
@ -147,4 +168,4 @@ passwd:
  users:
    - name: core
      ssh_authorized_keys:
-        - "${ssh_authorized_keys}"
+        - "${ssh_authorized_key}"
--- a/google-cloud/container-linux/kubernetes/controllers/controllers.tf
+++ b/google-cloud/container-linux/kubernetes/controllers/controllers.tf
@ -1,64 +1,43 @@
-# Managed Instance Group
-resource "google_compute_instance_group_manager" "controllers" {
-  name        = "${var.cluster_name}-controller-group"
-  description = "Compute instance group of ${var.cluster_name} controllers"
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "google_dns_record_set" "etcds" {
+  count = "${var.count}"

-  # Instance name prefix for instances in the group
-  base_instance_name = "${var.cluster_name}-controller"
-  instance_template  = "${google_compute_instance_template.controller.self_link}"
-  update_strategy    = "RESTART"
-  zone               = "${var.zone}"
-  target_size        = "${var.count}"
+  # DNS Zone name where record should be created
+  managed_zone = "${var.dns_zone_name}"

-  # Target pool instances in the group should be added into
-  target_pools = [
-    "${google_compute_target_pool.controllers.self_link}",
-  ]
+  # DNS record
+  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index,  var.dns_zone)}"
+  type = "A"
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  rrdatas = ["${element(google_compute_instance.controllers.*.network_interface.0.address, count.index)}"]
 }

-# Controller Container Linux Config
-data "template_file" "controller_config" {
-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
-
-  vars = {
-    k8s_dns_service_ip      = "${cidrhost(var.service_cidr, 10)}"
-    k8s_etcd_service_ip     = "${cidrhost(var.service_cidr, 15)}"
-    ssh_authorized_keys     = "${var.ssh_authorized_key}"
-    kubeconfig_ca_cert      = "${var.kubeconfig_ca_cert}"
-    kubeconfig_kubelet_cert = "${var.kubeconfig_kubelet_cert}"
-    kubeconfig_kubelet_key  = "${var.kubeconfig_kubelet_key}"
-    kubeconfig_server       = "${var.kubeconfig_server}"
-  }
+# Zones in the region
+data "google_compute_zones" "all" {
+  region = "${var.region}"
 }

-data "ct_config" "controller_ign" {
-  content      = "${data.template_file.controller_config.rendered}"
-  pretty_print = false
-}
+# Controller instances
+resource "google_compute_instance" "controllers" {
+  count = "${var.count}"

-resource "google_compute_instance_template" "controller" {
-  name_prefix  = "${var.cluster_name}-controller-"
-  description  = "Controller Instance template"
+  name         = "${var.cluster_name}-controller-${count.index}"
+  zone         = "${element(data.google_compute_zones.all.names, count.index)}"
  machine_type = "${var.machine_type}"

  metadata {
-    user-data = "${data.ct_config.controller_ign.rendered}"
+    user-data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
  }

-  scheduling {
-    automatic_restart = "${var.preemptible ? false : true}"
-    preemptible       = "${var.preemptible}"
-  }
+  boot_disk {
+    auto_delete = true

-  disk {
-    auto_delete  = true
-    boot         = true
-    source_image = "${var.os_image}"
-    disk_size_gb = "${var.disk_size}"
-
-    // Set explicit name to match the new default name set by the API.
-    // https://github.com/terraform-providers/terraform-provider-google/issues/574
-    device_name = "persistent-disk-0"
+    initialize_params {
+      image = "${var.os_image}"
+      size  = "${var.disk_size}"
+    }
  }

  network_interface {
@ -69,9 +48,46 @@ resource "google_compute_instance_template" "controller" {
  }

  can_ip_forward = true
+  tags = ["${var.cluster_name}-controller"]
+}

-  lifecycle {
-    # To update an Instance Template, Terraform should replace the existing resource
-    create_before_destroy = true
+# Controller Container Linux Config
+data "template_file" "controller_config" {
+  count = "${var.count}"
+
+  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster = "${join(",", formatlist("%s=https://%s:2380", null_resource.repeat.*.triggers.name, null_resource.repeat.*.triggers.domain))}"
+
+    k8s_dns_service_ip      = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix   = "${var.cluster_domain_suffix}"
+    ssh_authorized_key      = "${var.ssh_authorized_key}"
+    kubeconfig_ca_cert      = "${var.kubeconfig_ca_cert}"
+    kubeconfig_kubelet_cert = "${var.kubeconfig_kubelet_cert}"
+    kubeconfig_kubelet_key  = "${var.kubeconfig_kubelet_key}"
+    kubeconfig_server       = "${var.kubeconfig_server}"
  }
 }
+
+# Horrible hack to generate a Terraform list of a desired length without dependencies.
+# Ideal ${repeat("etcd", 3) -> ["etcd", "etcd", "etcd"]}
+resource null_resource "repeat" {
+  count = "${var.count}"
+
+  triggers {
+    name   = "etcd${count.index}"
+    domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+  }
+}
+
+data "ct_config" "controller_ign" {
+  count        = "${var.count}"
+  content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
+  pretty_print = false
+}
--- a/google-cloud/container-linux/kubernetes/controllers/network.tf
+++ b/google-cloud/container-linux/kubernetes/controllers/network.tf
@ -1,4 +1,9 @@
-# Controller Network Load balancer DNS record
+# Static IPv4 address for the Network Load Balancer
+resource "google_compute_address" "controllers-ip" {
+  name = "${var.cluster_name}-controllers-ip"
+}
+
+# DNS record for the Network Load Balancer
 resource "google_dns_record_set" "controllers" {
  # DNS Zone name where record should be created
  managed_zone = "${var.dns_zone_name}"
@ -12,12 +17,7 @@ resource "google_dns_record_set" "controllers" {
  rrdatas = ["${google_compute_address.controllers-ip.address}"]
 }

-# Static IP for the Network Load Balancer
-resource "google_compute_address" "controllers-ip" {
-  name = "${var.cluster_name}-controllers-ip"
-}
-
-# Network Load Balancer (i.e. forwarding rules)
+# Network Load Balancer (i.e. forwarding rule)
 resource "google_compute_forwarding_rule" "controller-https-rule" {
  name       = "${var.cluster_name}-controller-https-rule"
  ip_address = "${google_compute_address.controllers-ip.address}"
@ -25,26 +25,23 @@ resource "google_compute_forwarding_rule" "controller-https-rule" {
  target     = "${google_compute_target_pool.controllers.self_link}"
 }

-resource "google_compute_forwarding_rule" "controller-ssh-rule" {
-  name       = "${var.cluster_name}-controller-ssh-rule"
-  ip_address = "${google_compute_address.controllers-ip.address}"
-  port_range = "22"
-  target     = "${google_compute_target_pool.controllers.self_link}"
-}
-
-# Network Load Balancer target pool of instances.
+# Target pool of instances for the controller(s) Network Load Balancer
 resource "google_compute_target_pool" "controllers" {
  name = "${var.cluster_name}-controller-pool"

+  instances = [
+    "${google_compute_instance.controllers.*.self_link}",
+  ]
+
  health_checks = [
-    "${google_compute_http_health_check.ingress.name}",
+    "${google_compute_http_health_check.kubelet.name}",
  ]

  session_affinity = "NONE"
 }

 # Kubelet HTTP Health Check
-resource "google_compute_http_health_check" "ingress" {
+resource "google_compute_http_health_check" "kubelet" {
  name        = "${var.cluster_name}-kubelet-health"
  description = "Health check Kubelet health host port"

--- a/google-cloud/container-linux/kubernetes/controllers/outputs.tf
+++ b/google-cloud/container-linux/kubernetes/controllers/outputs.tf
@ -0,0 +1,7 @@
+output "etcd_fqdns" {
+  value = ["${null_resource.repeat.*.triggers.domain}"]
+}
+
+output "ipv4_public" {
+  value = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.assigned_nat_ip}"]
+}
--- a/google-cloud/container-linux/kubernetes/controllers/variables.tf
+++ b/google-cloud/container-linux/kubernetes/controllers/variables.tf
@ -30,9 +30,9 @@ variable "count" {
  description = "Number of controller compute instances the instance group should manage"
 }

-variable "zone" {
+variable "region" {
  type        = "string"
-  description = "Google zone that compute instances in the group should be created in (e.g. gcloud compute zones list)"
+  description = "Google Cloud region (e.g. us-central1, see `gcloud compute regions list`)."
 }

 variable "machine_type" {
@ -51,12 +51,6 @@ variable "disk_size" {
  description = "The size of the disk in gigabytes."
 }

-variable "preemptible" {
-  type        = "string"
-  default     = "false"
-  description = "If enabled, Compute Engine will terminate instances randomly within 24 hours"
-}
-
 // configuration

 variable "networking" {
@ -68,13 +62,19 @@ variable "networking" {
 variable "service_cidr" {
  description = <<EOD
 CIDR IP range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns, the 15th IP will be reserved for self-hosted etcd, and the 200th IP will be reserved for bootstrap self-hosted etcd.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
 EOD

  type    = "string"
  default = "10.3.0.0/16"
 }

+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
+
 // kubeconfig

 variable "kubeconfig_ca_cert" {
--- a/google-cloud/container-linux/kubernetes/network.tf
+++ b/google-cloud/container-linux/kubernetes/network.tf
@ -4,18 +4,6 @@ resource "google_compute_network" "network" {
  auto_create_subnetworks = true
 }

-resource "google_compute_firewall" "allow-ingress" {
-  name    = "${var.cluster_name}-allow-ingress"
-  network = "${google_compute_network.network.name}"
-
-  allow {
-    protocol = "tcp"
-    ports    = [80, 443]
-  }
-
-  source_ranges = ["0.0.0.0/0"]
-}
-
 resource "google_compute_firewall" "allow-ssh" {
  name    = "${var.cluster_name}-allow-ssh"
  network = "${google_compute_network.network.name}"
@ -26,31 +14,55 @@ resource "google_compute_firewall" "allow-ssh" {
  }

  source_ranges = ["0.0.0.0/0"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }

-resource "google_compute_firewall" "allow-internal" {
-  name    = "${var.cluster_name}-allow-internal"
+resource "google_compute_firewall" "allow-apiserver" {
+  name    = "${var.cluster_name}-allow-apiserver"
  network = "${google_compute_network.network.name}"

  allow {
    protocol = "tcp"
-    ports    = ["1-65535"]
+    ports    = [443]
  }

+  source_ranges = ["0.0.0.0/0"]
+  target_tags = ["${var.cluster_name}-controller"]
+}
+
+
+resource "google_compute_firewall" "allow-ingress" {
+  name    = "${var.cluster_name}-allow-ingress"
+  network = "${google_compute_network.network.name}"
+
  allow {
-    protocol = "udp"
-    ports    = ["1-65535"]
+    protocol = "tcp"
+    ports    = [80, 443]
  }

-  source_ranges = ["10.0.0.0/8"]
+  source_ranges = ["0.0.0.0/0"]
+  target_tags = ["${var.cluster_name}-worker"]
+}
+
+resource "google_compute_firewall" "internal-etcd" {
+  name    = "${var.cluster_name}-internal-etcd"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "tcp"
+    ports    = [2380]
+  }
+
+  source_tags = ["${var.cluster_name}-controller"]
+  target_tags = ["${var.cluster_name}-controller"]
 }

 # Calico BGP and IPIP
 # https://docs.projectcalico.org/v2.5/reference/public-cloud/gce
-resource "google_compute_firewall" "allow-calico" {
+resource "google_compute_firewall" "internal-calico" {
  count = "${var.networking == "calico" ? 1 : 0}"

-  name    = "${var.cluster_name}-allow-calico"
+  name    = "${var.cluster_name}-internal-calico"
  network = "${google_compute_network.network.name}"

  allow {
@ -62,5 +74,63 @@ resource "google_compute_firewall" "allow-calico" {
    protocol = "ipip"
  }

-  source_ranges = ["10.0.0.0/8"]
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# flannel
+resource "google_compute_firewall" "internal-flannel" {
+  count = "${var.networking == "flannel" ? 1 : 0}"
+
+  name    = "${var.cluster_name}-internal-flannel"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "udp"
+    ports    = [8472]
+  }
+
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# Allow prometheus (workload) to scrape node-exporter daemonset
+resource "google_compute_firewall" "internal-node-exporter" {
+  name    = "${var.cluster_name}-internal-node-exporter"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "tcp"
+    ports    = [9100]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# kubelet API to allow kubectl exec and log
+resource "google_compute_firewall" "internal-kubelet" {
+  name    = "${var.cluster_name}-internal-kubelet"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "tcp"
+    ports    = [10250]
+  }
+
+  source_tags = ["${var.cluster_name}-controller"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+resource "google_compute_firewall" "internal-kubelet-readonly" {
+  name    = "${var.cluster_name}-internal-kubelet-readonly"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "tcp"
+    ports    = [10255]
+  }
+
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }
--- a/google-cloud/container-linux/kubernetes/outputs.tf
+++ b/google-cloud/container-linux/kubernetes/outputs.tf
@ -1,3 +1,15 @@
+output "controllers_ipv4_public" {
+  value = ["${module.controllers.ipv4_public}"]
+}
+
 output "ingress_static_ip" {
  value = "${module.workers.ingress_static_ip}"
 }
+
+output "network_name" {
+  value = "${google_compute_network.network.name}"
+}
+
+output "network_self_link" {
+  value = "${google_compute_network.network.self_link}"
+}
--- a/google-cloud/container-linux/kubernetes/require.tf
+++ b/google-cloud/container-linux/kubernetes/require.tf
@ -5,7 +5,7 @@ terraform {
 }

 provider "google" {
-  version = "~> 1.0"
+  version = "~> 1.2"
 }

 provider "local" {
--- a/google-cloud/container-linux/kubernetes/ssh.tf
+++ b/google-cloud/container-linux/kubernetes/ssh.tf
@ -1,12 +1,80 @@
+# Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
+resource "null_resource" "copy-secrets" {
+  depends_on = ["module.bootkube"]
+  count      = "${var.controller_count}"
+
+  connection {
+    type    = "ssh"
+    host    = "${element(module.controllers.ipv4_public, count.index)}"
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.kubeconfig}"
+    destination = "$HOME/kubeconfig"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_ca_cert}"
+    destination = "$HOME/etcd-client-ca.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_cert}"
+    destination = "$HOME/etcd-client.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_key}"
+    destination = "$HOME/etcd-client.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_cert}"
+    destination = "$HOME/etcd-server.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_key}"
+    destination = "$HOME/etcd-server.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_cert}"
+    destination = "$HOME/etcd-peer.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_key}"
+    destination = "$HOME/etcd-peer.key"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mkdir -p /etc/ssl/etcd/etcd",
+      "sudo mv etcd-client* /etc/ssl/etcd/",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
+      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
+      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
+      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
+      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
+      "sudo chown -R etcd:etcd /etc/ssl/etcd",
+      "sudo chmod -R 500 /etc/ssl/etcd",
+      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
+    ]
+  }
+}
+
 # Secure copy bootkube assets to ONE controller and start bootkube to perform
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
-  depends_on = ["module.controllers", "module.workers", "module.bootkube"]
+  depends_on = ["module.controllers", "module.bootkube", "module.workers", "null_resource.copy-secrets"]

-  # TODO: SSH to a controller's IP instead of waiting on DNS resolution
  connection {
    type    = "ssh"
-    host    = "${format("%s.%s", var.cluster_name, var.dns_zone)}"
+    host    = "${element(module.controllers.ipv4_public, 0)}"
    user    = "core"
    timeout = "15m"
  }
--- a/google-cloud/container-linux/kubernetes/variables.tf
+++ b/google-cloud/container-linux/kubernetes/variables.tf
@ -3,9 +3,9 @@ variable "cluster_name" {
  description = "Cluster name"
 }

-variable "zone" {
+variable "region" {
  type        = "string"
-  description = "Google Cloud zone (e.g. us-central1-f, see `gcloud compute zones list`)"
+  description = "Google Cloud Region (e.g. us-central1, see `gcloud compute regions list`)"
 }

 variable "dns_zone" {
@ -46,12 +46,6 @@ variable "worker_count" {
  description = "Number of workers"
 }

-variable "controller_preemptible" {
-  type        = "string"
-  default     = "false"
-  description = "If enabled, Compute Engine will terminate controllers randomly within 24 hours"
-}
-
 variable "worker_preemptible" {
  type        = "string"
  default     = "false"
@ -80,9 +74,15 @@ variable "pod_cidr" {
 variable "service_cidr" {
  description = <<EOD
 CIDR IP range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns, the 15th IP will be reserved for self-hosted etcd, and the 200th IP will be reserved for bootstrap self-hosted etcd.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
 EOD

  type    = "string"
  default = "10.3.0.0/16"
 }
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
--- a/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -50,15 +50,14 @@ systemd:
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain=cluster.local \
+          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --require-kubeconfig
+          --pod-manifest-path=/etc/kubernetes/manifests
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
        RestartSec=5
@ -105,7 +104,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
-          KUBELET_IMAGE_TAG=v1.8.2
+          KUBELET_IMAGE_TAG=v1.8.6
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -123,7 +122,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.8.2 \
+            docker://gcr.io/google_containers/hyperkube:v1.8.6 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/google-cloud/container-linux/kubernetes/workers/network.tf
+++ b/google-cloud/container-linux/kubernetes/workers/network.tf
@ -1,4 +1,4 @@
-# Static IP for the Network Load Balancer
+# Static IPv4 address for the Network Load Balancer
 resource "google_compute_address" "ingress-ip" {
  name = "${var.cluster_name}-ingress-ip"
 }
--- a/google-cloud/container-linux/kubernetes/workers/variables.tf
+++ b/google-cloud/container-linux/kubernetes/workers/variables.tf
@ -20,9 +20,9 @@ variable "count" {
  description = "Number of worker compute instances the instance group should manage"
 }

-variable "zone" {
+variable "region" {
  type        = "string"
-  description = "Google zone that compute instances in the group should be created in (e.g. gcloud compute zones list)"
+  description = "Google Cloud region (e.g. us-central1, see `gcloud compute regions list`)."
 }

 variable "machine_type" {
@ -52,13 +52,19 @@ variable "preemptible" {
 variable "service_cidr" {
  description = <<EOD
 CIDR IP range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns, the 15th IP will be reserved for self-hosted etcd, and the 200th IP will be reserved for bootstrap self-hosted etcd.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
 EOD

  type    = "string"
  default = "10.3.0.0/16"
 }

+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}
+
 # kubeconfig

 variable "kubeconfig_ca_cert" {
--- a/google-cloud/container-linux/kubernetes/workers/workers.tf
+++ b/google-cloud/container-linux/kubernetes/workers/workers.tf
@ -1,16 +1,17 @@
-# Managed Instance Group
-resource "google_compute_instance_group_manager" "workers" {
+# Regional managed instance group maintains a homogeneous set of workers that
+# span the zones in the region.
+resource "google_compute_region_instance_group_manager" "workers" {
  name        = "${var.cluster_name}-worker-group"
  description = "Compute instance group of ${var.cluster_name} workers"

-  # Instance name prefix for instances in the group
+  # instance name prefix for instances in the group
  base_instance_name = "${var.cluster_name}-worker"
  instance_template  = "${google_compute_instance_template.worker.self_link}"
-  update_strategy    = "RESTART"
-  zone               = "${var.zone}"
-  target_size        = "${var.count}"
+  region             = "${var.region}"

-  # Target pool instances in the group should be added into
+  target_size = "${var.count}"
+
+  # target pool to which instances in the group should be added
  target_pools = [
    "${google_compute_target_pool.workers.self_link}",
  ]
@ -23,6 +24,7 @@ data "template_file" "worker_config" {
  vars = {
    k8s_dns_service_ip      = "${cidrhost(var.service_cidr, 10)}"
    k8s_etcd_service_ip     = "${cidrhost(var.service_cidr, 15)}"
+    cluster_domain_suffix   = "${var.cluster_domain_suffix}"
    ssh_authorized_key      = "${var.ssh_authorized_key}"
    kubeconfig_ca_cert      = "${var.kubeconfig_ca_cert}"
    kubeconfig_kubelet_cert = "${var.kubeconfig_kubelet_cert}"
@ -55,10 +57,6 @@ resource "google_compute_instance_template" "worker" {
    boot         = true
    source_image = "${var.os_image}"
    disk_size_gb = "${var.disk_size}"
-
-    // Set explicit name to match the new default name set by the API.
-    // https://github.com/terraform-providers/terraform-provider-google/issues/574
-    device_name = "persistent-disk-0"
  }

  network_interface {
@ -70,7 +68,7 @@ resource "google_compute_instance_template" "worker" {

  can_ip_forward = true

-  tags = ["worker"]
+  tags = ["worker", "${var.cluster_name}-worker"]

  lifecycle {
    # To update an Instance Template, Terraform should replace the existing resource
Author	SHA1	Message	Date
Dalton Hubble	201a38bd90	Update CHANGELOG.md for v1.8.6	2017-12-22 13:00:18 -08:00
Dalton Hubble	fbdd946601	Update Kubernetes from v1.8.5 to v1.8.6	2017-12-21 11:20:37 -08:00
Barak Michener	19102636a9	Add link to dashboard 315	2017-12-15 18:52:40 -08:00
Dalton Hubble	21e540159b	addons: Update grafana from v4.6.2 to v4.6.3 * https://github.com/grafana/grafana/releases/tag/v4.6.3	2017-12-15 16:09:14 -08:00
Dalton Hubble	43e65a4d13	Update CHANGELOG.md for v1.8.5	2017-12-15 02:04:13 -08:00
Barak Michener	e79088baa0	Add optional cluster_domain_suffix variable * Allow kube-dns to respond to DNS queries with a custom suffix, instead of the default 'cluster.local' * Useful when multiple clusters exist on the same local network and wish to query services on one another	2017-12-15 01:45:52 -08:00
Dalton Hubble	495e33e213	Update bootkube and terraform-render-bootkube to v0.9.1	2017-12-15 01:45:02 -08:00
Dalton Hubble	63f5a26a72	Eliminate steps to move self-hosted etcd assets * bootkube/assets/experimental/* assets corresponded to self-hosted etcd manifests, which are no longer an option in Typhoon	2017-12-13 01:06:56 -08:00
Lars Fenneberg	eea79e895d	Fix manifest consolidation in bootkube start wrapper * Fix manifest existence test in /opt/bootkube/bootkube-start to also work with more than one directory	2017-12-12 23:08:22 -08:00
Dalton Hubble	99c07661c6	Fix old Container Linux versions mentioned in docs	2017-12-11 23:36:16 -08:00
Dalton Hubble	521a1f0fee	addons: Update heapster from v1.4.3 to v1.5.0 * Rollback addon-resizer to 1.7 to address issues in large clusters https://github.com/kubernetes/kubernetes/pull/52536	2017-12-11 23:34:25 -08:00
Dalton Hubble	7345cb6419	addons: Update nginx-ingress to 0.9.0	2017-12-11 00:48:15 -08:00
Dalton Hubble	a481d71d7d	addons: Update nginx-ingress to 0.9.0-beta.19 * Undo rollback `f00ecde854` * Port binding regression only occurs with --enable-ssl-passthrough, which isn't used in these examples. See https://github.com/kubernetes/ingress-nginx/issues/1788	2017-12-11 00:44:32 -08:00
Dalton Hubble	831a5c976c	Add Kubernetes Dashboard warning and improve changelog	2017-12-09 22:38:27 -08:00
Dalton Hubble	85e6783503	Recommend Container Linux images with Docker 17.09 * Container Linux stable and beta now provide Docker 17.09 (instead of 1.12). Recommend images which provide 17.09. * Older clusters (with CLUO addon) auto-update node's Container Linux version and will begin using Docker 17.09.	2017-12-09 22:14:13 -08:00
Dalton Hubble	165396d6aa	Update Kubernetes from v1.8.4 to v1.8.5	2017-12-09 21:28:31 -08:00
Vincent Palmer	ce49a93d5d	Fix issue with etcd-member failing to resolve peers * When restarting masters, `etcd-member.service` may fail to lookup peers if /etc/resolv.conf hasn't been populated yet. Require the wait-for-dns.service.	2017-12-09 20:12:49 -08:00
Khris Richardson	e623439eec	Fix typos in docs and CONTRIBUTING.md	2017-12-09 19:58:09 -08:00
Dalton Hubble	9548572d98	Add kubelet --volume-plugin-dir flag on bare-metal * Kubelet will search path for flexvolume plugins	2017-12-05 13:12:53 -08:00
Dalton Hubble	f00ecde854	Rollback nginx-ingress on GCE to 0.9.0-beta.17 * https://github.com/kubernetes/ingress-nginx/issues/1788	2017-12-02 14:06:22 -08:00
Dalton Hubble	d85300f947	Clarify only Terraform v0.10.x should be used * It is not safe to update to Terraform v0.11.x yet * https://github.com/hashicorp/terraform/issues/16824	2017-12-02 01:31:39 -08:00
Dalton Hubble	65f006e6cc	addons: Sync prometheus alerts to upstream * https://github.com/coreos/prometheus-operator/pull/774	2017-12-01 23:24:08 -08:00
Dalton Hubble	8d3817e0ae	addons: Update nginx-ingress to 0.9.0-beta.19 * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.9.0-beta.19	2017-12-01 22:32:33 -08:00
Dalton Hubble	5f5eec1175	Update bootkube and terraform-render-bootkube to v0.9.0	2017-12-01 22:27:48 -08:00
Dalton Hubble	5308fde3d3	Add Kubernetes certification badge	2017-11-29 19:26:49 -08:00
Dalton Hubble	9ab61d7bf5	Add Typhoon images with and without text * Serve images from GCS poseidon, rather than dghubble	2017-11-29 01:01:01 -08:00
Dalton Hubble	6483f613c5	Update Kubernetes from v1.8.3 to v1.8.4	2017-11-28 21:52:11 -08:00
Dalton Hubble	56c6bf431a	Update terraform-render-bootkube for Kubernetes v1.8.4 * Update hyperkube from v1.8.3 to v1.8.4 * Remove flock from bootstrap-apiserver and kube-apiserver * Remove unused critical-pod annotations in manifests * Use service accounts for kube-proxy and pod-checkpointer * Update Calico from v2.6.1 to v2.6.3 * Update flannel from v0.9.0 to v0.9.1 * Remove Calico termination grace period to prevent calico from getting stuck for extended periods * https://github.com/poseidon/terraform-render-bootkube/pull/29	2017-11-28 21:42:26 -08:00
Dalton Hubble	63ab117205	addons: Add prometheus rules for DaemonSets * https://github.com/coreos/prometheus-operator/pull/755	2017-11-16 23:51:21 -08:00
Dalton Hubble	1cd262e712	addons: Fix prometheus K8SApiServerLatency alert rule * https://github.com/coreos/prometheus-operator/issues/751	2017-11-16 23:37:15 -08:00
Dalton Hubble	32bdda1b6c	addons: Update Grafana from v4.6.1 to v4.6.2 * https://github.com/grafana/grafana/releases/tag/v4.6.2	2017-11-16 23:34:36 -08:00
Dalton Hubble	07d257aa7b	Add initrd kernel argument needed by UEFI clients * https://github.com/coreos/bugs/issues/1239	2017-11-16 23:19:51 -08:00
Dalton Hubble	fd96067125	Fix docs link for security issue reporting	2017-11-10 21:38:41 -08:00
Dalton Hubble	9d16f5c78a	Update min Google plugin and remove target pool workaround * With google provider 1.2, target pool instances can use self_link and zone/name formats without causing a diff on each plan * Original workaround: `77fc14db71`	2017-11-10 21:15:19 -08:00
Dalton Hubble	159443bae7	addons: Add better alerting rules to Prometheus manifests * Adapt the coreos/prometheus-operator alerting rules for Typhoon, https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests * Add controller manager and scheduler shim services to let prometheus discover them via service endpoints * Fix several alert rules to use service endpoint discovery * A few rules still don't do much, but they default to green	2017-11-10 20:57:47 -08:00
Dalton Hubble	119dc859d3	addons: Update nginx-ingress to 0.9.0-beta.17 * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.9.0-beta.17	2017-11-10 20:16:40 -08:00
Dalton Hubble	5f6b0728c5	Update bootkube and terraform-render-bootkube to v0.8.2	2017-11-10 20:01:37 -08:00
Dalton Hubble	d774c51297	Update Kubernetes from v1.8.2 to v1.8.3	2017-11-08 23:34:19 -08:00
Dalton Hubble	f6a8fb363e	Remove deprecated kubelet --require-kubeconfig flag * https://github.com/kubernetes/kubernetes/pull/40050	2017-11-08 23:34:19 -08:00
Dalton Hubble	f570af9418	addons: Update from Prometheus v1.8.2 to v2.0.0	2017-11-08 22:48:23 -08:00
Dalton Hubble	4ec6732b98	Output the Google network name and self_link * Allow users to add custom firewall rules for unique cases	2017-11-08 00:19:49 -08:00
Dalton Hubble	ea1efb536a	Remove old firewall rule for bootstrap self-hosted etcd	2017-11-08 00:15:20 -08:00
Dalton Hubble	451fd86470	Improve internal firewall rules on Google Cloud * Whitelist internal traffic between controllers and workers * Switch to tag-based firewall policies rather than source IP	2017-11-08 00:15:06 -08:00
Dalton Hubble	b1b611b22c	Add docs to use one controller on Google Cloud	2017-11-07 19:51:03 -08:00
Dalton Hubble	eabf00fbf1	Add missing controller dependency before bootkube start * Require the controller module to be completed before starting to remote exec bootkube start, otherwise its possible the controller nodes were created, but not the network load balancer	2017-11-07 19:12:05 -08:00
Dalton Hubble	8eaa72c1ca	addons: Update nginx-ingress to 0.9.0-beta.16 * Image registry changed from gcr.io to quay.io * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.9.0-beta.16	2017-11-06 23:15:15 -08:00
Dalton Hubble	58cf82da56	Promote AWS platform from alpha to beta	2017-11-06 21:38:24 -08:00
Dalton Hubble	ccc832f468	Add firewall rule to allow apiserver to proxy other controller kubelets * Prometheus proxies through the apiserver to scrape kubelets * In multi-controller setups, an apiserver must be able to scrape kubelets (10250) on other controllers	2017-11-06 01:03:53 -08:00
Dalton Hubble	90f8d62204	Add firewall rules to allow prometheus to reach node-exporter * node_exporter service endpoints run on hostNetwork port 9100 * Re-evaluate after https://github.com/kubernetes-incubator/bootkube/pull/711	2017-11-06 01:03:53 -08:00
Dalton Hubble	af5c413abf	Focus controller ELB on load balancing apiservers * ELB distributing load across controllers is no longer the mechanism used to SSH to instances to distribute secrets * Focus the ELB on load balancing across apiserver and edit the HTTP health check to an SSL:443 check	2017-11-06 01:03:53 -08:00
Dalton Hubble	168c487484	Remove mention of self-hosted etcd, its deprecated	2017-11-06 01:03:53 -08:00
Dalton Hubble	805dd772a8	Run etcd cluster on-host, across controllers on AWS * Change controllers ASG to heterogeneous EC2 instances * Create DNS records for each controller's private IP for etcd * Change etcd to run on-host, across controllers (etcd-member.service) * Reduce time to bootstrap a cluster * Deprecate self-hosted-etcd on the AWS platform	2017-11-06 01:03:53 -08:00
Dalton Hubble	c6ec6596d8	Minor cleanup for zones, docs, and outputs * Spread across all zones, regardless of UP/DOWN state * Remove unused outputs of private IPs	2017-11-06 00:56:26 -08:00
Dalton Hubble	47a9989927	Fix null_resource ordering constraints * Ensure etcd TLS assets and kubeconfig are copied before any attempt is made to run bootkube start	2017-11-06 00:55:44 -08:00
Dalton Hubble	10b977d54a	addons: Set kube-state-metrics to have clusterIP None * kube-state-metrics service exists to facilitate prometheus discovery	2017-11-05 17:54:09 -08:00
Dalton Hubble	b7a268fc45	addons: Add prometheus alertmanager flag * Pass -alertmanager.url to work with a user's in-cluster alertmanager deployment, if any	2017-11-05 15:50:46 -08:00
Dalton Hubble	279f36effd	addons: Add grafana 4.6.1 and extend prometheus docs	2017-11-05 15:23:56 -08:00
Dalton Hubble	77fc14db71	Workaround target pool issue by listing instances as zone/name * Instances can be listed by zone/name or self_link URL, but the provider desires they be in zone/name form, which causes a diff * https://github.com/terraform-providers/terraform-provider-google/issues/46	2017-11-05 14:07:05 -08:00
Dalton Hubble	2b0296d671	Create controller instances across zones in the region * Change controller instances to automatically span zones in a region * Remove the `zone` required variable	2017-11-05 13:24:32 -08:00
Dalton Hubble	7b38271212	Run etcd cluster on-host, across controllers on Google Cloud * Change controllers from a managed group to individual instances * Create discrete DNS records to each controller's private IP for etcd * Change etcd to run on-host, across controllers (etcd-member.service) * Reduce time to bootstrap a cluster * Deprecate self-hosted-etcd on the Google Cloud platform	2017-11-05 11:03:35 -08:00
Dalton Hubble	ae07a21e3d	addons: Omit static resource requests/limits for kube-state-metrics * Allow the addon-resizer to dynamically set resource values * https://github.com/kubernetes/kube-state-metrics/pull/285	2017-11-04 14:41:04 -07:00
Dalton Hubble	0ab1ae3210	addons: Fix typo in kube-state-metrics strategy	2017-11-04 14:39:56 -07:00
Dalton Hubble	67e3d2b86e	docs: GCE network bandwidth is excellent, even btw zones * Remove performance note that the GCE vs AWS network performance is not an equal comparison. On both platforms, workers now span the (availability) zones of a region. * Testing host-to-host and pod-to-pod network bandwidth between nodes (now located in different zones) showed no reduction in bandwidth	2017-11-04 14:08:20 -07:00
Dalton Hubble	a48dd9ebd8	Require google provider version ~> 1.1 * Require google provider plugin 1.1 or higher which includes fix: https://github.com/terraform-providers/terraform-provider-google/issues/574 * Remove workaround which statically set the persistent disk name * Original reasons for workaround in `a97df839` or GH #34	2017-11-04 12:59:19 -07:00
Dalton Hubble	26a291aef4	Remove controller_preemptible option on Google Cloud * Controller preemption is not safe or covered in documentation. Delete the option, the variable is a holdover from old experiments * Note, worker_preemeptible is still a great feature that's supported	2017-11-04 12:59:19 -07:00
Dalton Hubble	251a14519f	Fix typo in internal template variable name * ssh_authorized_keys should be ssh_authorized_key to match the user facing variable which only allows a single SSH authorized key	2017-11-04 12:59:19 -07:00
Dalton Hubble	6300383b43	Change worker managed instance group to span zones in region * Change Google Cloud module to require the `region` variable * Workers are created in random zones within the given region * Tolerate Google Cloud zone failures or capacity issues * If workers are preempted (if enabled), replacement instances can be drawn from any zone in the region, which should avoid scheduling issues that were possible before if a single zone aggressively preempts instances (presumably due to Google Cloud capacity)	2017-11-04 12:59:19 -07:00
Dalton Hubble	e32885c9cd	addons: Update prometheus from v1.8.0 to v1.8.2 * https://github.com/prometheus/prometheus/releases/tag/v1.8.2	2017-11-04 11:00:39 -07:00
Dalton Hubble	fe8afdbee9	Update Typhoon logo and favicon	2017-11-04 01:20:17 -07:00