Improve the issue template prompt

* Clarify that reporting versions as latest is not
helpful

.github/ISSUE_TEMPLATE.md

@@ -4,11 +4,11 @@
 
 ### Environment
 
-* Platform: aws, bare-metal, google-cloud, digital-ocean
+* Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, fedora-cloud
+* OS: container-linux, fedora-atomic
-* Terraform: `terraform version`
+* Ref: Release version or Git SHA (reporting latest is **not** helpful)
-* Plugins: Provider plugin versions
+* Terraform: `terraform version` (reporting latest is **not** helpful)
-* Ref: Git SHA (if applicable)
+* Plugins: Provider plugin versions (reporting latest is **not** helpful)
 
 ### Problem
 

CHANGES.md

@@ -4,6 +4,319 @@ Notable changes between versions.
 
 ## Latest
 
+## v1.11.3
+
+* Kubernetes [v1.11.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113)
+* Introduce Typhoon for Azure as alpha ([#288](https://github.com/poseidon/typhoon/pull/288))
+  * Special thanks @justaugustus for an earlier variant
+* Update Calico from v3.1.3 to v3.2.1 ([#278](https://github.com/poseidon/typhoon/pull/278))
+
+#### AWS
+
+* Remove firewall rule allowing ICMP packets to nodes ([#285](https://github.com/poseidon/typhoon/pull/285))
+
+#### Bare-Metal
+
+* Remove `controller_networkds` and `worker_networkds` variables. Use Container Linux Config snippets [#277](https://github.com/poseidon/typhoon/pull/277)
+
+#### Google Cloud
+
+* Fix firewall to allow etcd client port 2379 traffic between controller nodes ([#287](https://github.com/poseidon/typhoon/pull/287))
+  * kube-apiservers were only able to connect to their node's local etcd peer. While master node outages were tolerated, reaching a healthy peer took longer than neccessary in some cases
+  * Reduce time needed to bootstrap the cluster
+* Remove firewall rule allowing workers to access Nginx Ingress health check ([#284](https://github.com/poseidon/typhoon/pull/284))
+  * Nginx Ingress addon no longer uses hostNetwork, Prometheus scrapes via CNI network
+
+#### Addons
+
+* Update nginx-ingress from 0.17.1 to 0.19.0
+* Update kube-state-metrics from v1.3.1 to v1.4.0
+* Update Grafana from 5.2.2 to 5.2.4
+
+## v1.11.2
+
+* Kubernetes [v1.11.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1112)
+* Update etcd from v3.3.8 to [v3.3.9](https://github.com/coreos/etcd/blob/master/CHANGELOG-3.3.md#v339-2018-07-24)
+* Use kubernetes-incubator/bootkube v0.13.0
+* Fix Fedora Atomic modules' Kubelet version ([#270](https://github.com/poseidon/typhoon/issues/270))
+
+#### Bare-Metal
+
+* Introduce [Container Linux Config snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) on bare-metal
+  * Validate and additively merge custom Container Linux Configs during terraform plan
+  * Define files, systemd units, dropins, networkd configs, mounts, users, and more
+  * [Require](https://typhoon.psdn.io/cl/bare-metal/#terraform-setup) `terraform-provider-ct` plugin v0.2.1 (**action required!**)
+
+#### Addons
+
+* Update nginx-ingress from 0.16.2 to 0.17.1
+* Add nginx-ingress manifests for bare-metal
+* Update Grafana from 5.2.1 to 5.2.2
+* Update heapster from v1.5.3 to v1.5.4
+
+## v1.11.1
+
+* Kubernetes [v1.11.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1111)
+
+#### Addons
+
+* Update Prometheus from v2.3.1 to v2.3.2
+
+#### Errata
+
+* Fedora Atomic modules shipped with Kubelet v1.11.0, instead of v1.11.1. Fixed in [#270](https://github.com/poseidon/typhoon/issues/270).
+
+## v1.11.0
+
+* Kubernetes [v1.11.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1110)
+* Force apiserver to stop listening on `127.0.0.1:8080`
+* Replace `kube-dns` with [CoreDNS](https://coredns.io/) ([#261](https://github.com/poseidon/typhoon/pull/261))
+  * Edit the `coredns` ConfigMap to [customize](https://coredns.io/plugins/)
+  * CoreDNS doesn't use a resizer. For large clusters, scaling may be required. 
+
+#### AWS
+
+* Update from Fedora Atomic 27 to 28 ([#258](https://github.com/poseidon/typhoon/pull/258))
+
+#### Bare-Metal
+
+* Update from Fedora Atomic 27 to 28 ([#263](https://github.com/poseidon/typhoon/pull/263))
+
+#### Google
+
+* Promote Google Cloud to stable
+* Update from Fedora Atomic 27 to 28 ([#259](https://github.com/poseidon/typhoon/pull/259))
+* Remove `ingress_static_ip` module output. Use `ingress_static_ipv4`.
+* Remove `controllers_ipv4_public` module output.
+
+#### Addons
+
+* Update nginx-ingress from 0.15.0 to 0.16.2
+* Update Grafana from 5.1.4 to [5.2.1](http://docs.grafana.org/guides/whats-new-in-v5-2/)
+* Update heapster from v1.5.2 to v1.5.3
+
+## v1.10.5
+
+* Kubernetes [v1.10.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1105)
+* Update etcd from v3.3.6 to v3.3.8 ([#243](https://github.com/poseidon/typhoon/pull/243), [#247](https://github.com/poseidon/typhoon/pull/247))
+
+#### AWS
+
+* Switch `kube-apiserver` port from 443 to 6443 ([#248](https://github.com/poseidon/typhoon/pull/248))
+* Combine apiserver and ingress NLBs ([#249](https://github.com/poseidon/typhoon/pull/249))
+  * Reduce cost by ~$18/month per cluster. Typhoon AWS clusters now use one network load balancer.
+  * Ingress addon users may keep using CNAME records to the `ingress_dns_name` module output (few million RPS)
+  * Ingress users with heavy traffic (many million RPS) should create a separate NLB(s)
+* Worker pools no longer include an extraneous load balancer. Remove worker module's `ingress_dns_name` output
+* Disable detailed (paid) monitoring on worker nodes ([#251](https://github.com/poseidon/typhoon/pull/251))
+  * Favor Prometheus for cloud-agnostic metrics, aggregation, and alerting
+* Add `worker_target_group_http` and `worker_target_group_https` module outputs to allow custom load balancing
+* Add `target_group_http` and `target_group_https` worker module outputs to allow custom load balancing
+
+#### Bare-Metal
+
+* Switch `kube-apiserver` port from 443 to 6443 ([#248](https://github.com/poseidon/typhoon/pull/248))
+  * Users who exposed kube-apiserver on a WAN via their router/load-balancer will need to adjust its configuration (e.g. DNAT 6443). Most apiservers are on a LAN (internal, VPN-only, etc) so if you didn't specially configure network gear for 443, no change is needed. (possible action required)
+* Fix possible deadlock when provisioning clusters larger than 10 nodes ([#244](https://github.com/poseidon/typhoon/pull/244)) 
+
+#### DigitalOcean
+
+* Switch `kube-apiserver` port from 443 to 6443 ([#248](https://github.com/poseidon/typhoon/pull/248))
+  * Update firewall rules and generated kubeconfig's
+
+#### Google Cloud
+
+* Use global HTTP and TCP proxy load balancing for Kubernetes Ingress ([#252](https://github.com/poseidon/typhoon/pull/252))
+  * Switch Ingress from regional network load balancers to global HTTP/TCP Proxy load balancing
+  * Reduce cost by ~$19/month per cluster. Google bills the first 5 global and regional forwarding rules separately. Typhoon clusters now use 3 global and 0 regional forwarding rules.
+* Worker pools no longer include an extraneous load balancer. Remove worker module's `ingress_static_ip` output
+* Allow using nginx-ingress addon on Fedora Atomic clusters ([#200](https://github.com/poseidon/typhoon/issues/200))
+* Add `worker_instance_group` module output to allow custom global load balancing
+* Add `instance_group` worker module output to allow custom global load balancing
+* Deprecate `ingress_static_ip` module output. Add `ingress_static_ipv4` module output instead.
+* Deprecate `controllers_ipv4_public` module output
+
+#### Addons
+
+* Update CLUO from v0.6.0 to v0.7.0 ([#242](https://github.com/poseidon/typhoon/pull/242))
+* Update Prometheus from v2.3.0 to v2.3.1
+* Update Grafana from 5.1.3 to 5.1.4
+* Drop `hostNetwork` from nginx-ingress addon
+  * Both flannel and Calico support host port via `portmap`
+  * Allows writing NetworkPolicies that reference ingress pods in `from` or `to`. HostNetwork pods were difficult to write network policy for since they could circumvent the CNI network to communicate with pods on the same node.
+
+## v1.10.4
+
+* Kubernetes [v1.10.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1104)
+* Update etcd from v3.3.5 to v3.3.6
+* Update Calico from v3.1.2 to v3.1.3
+
+#### Addons
+
+* Update Prometheus from v2.2.1 to v2.3.0
+* Add Prometheus liveness and readiness probes
+* Annotate Grafana service so Prometheus scrapes metrics
+* Label namespaces to ease writing Network Policies
+
+## v1.10.3
+
+* Kubernetes [v1.10.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1103)
+* Add [Flatcar Linux](https://docs.flatcar-linux.org/) (Container Linux derivative) as an option for AWS and bare-metal (thanks @kinvolk folks)
+* Allow bearer token authentication to the Kubelet ([#216](https://github.com/poseidon/typhoon/issues/216))
+  * Require Webhook authorization to the Kubelet
+  * Switch apiserver X509 client cert org to satisfy new authorization requirement
+* Require Terraform v0.11.x and drop support for v0.10.x ([migration guide](https://typhoon.psdn.io/topics/maintenance/#terraform-v011x))
+* Update etcd from v3.3.4 to v3.3.5 ([#213](https://github.com/poseidon/typhoon/pull/213))
+* Update Calico from v3.1.1 to v3.1.2
+
+#### AWS
+
+* Allow Flatcar Linux by setting `os_image` to flatcar-stable (default), flatcar-beta, flatcar-alpha ([#211](https://github.com/poseidon/typhoon/pull/211))
+* Replace `os_channel` variable with `os_image` to align naming across clouds
+  * Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (**action required!**)
+* Allow preemptible workers via spot instances ([#202](https://github.com/poseidon/typhoon/pull/202))
+  * Add `worker_price` to allow worker spot instances. Default to empty string for the worker autoscaling group to use regular on-demand instances
+  * Add `spot_price` to internal `workers` module for spot [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+
+#### Bare-Metal
+
+* Allow Flatcar Linux by setting `os_channel` to flatcar-stable, flatcar-beta, flatcar-alpha ([#220](https://github.com/poseidon/typhoon/pull/220))
+* Replace `container_linux_channel` variable with `os_channel`
+  * Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (**action required!**)
+* Replace `container_linux_version` variable with `os_version`
+* Add `network_ip_autodetection_method` variable for Calico host IPv4 address detection
+  * Use Calico's default "first-found" to support single NIC and bonded NIC nodes 
+  * Allow [alternative](https://docs.projectcalico.org/v3.1/reference/node/configuration#ip-autodetection-methods) methods for multi NIC nodes, like can-reach=IP or interface=REGEX
+* Deprecate `container_linux_oem` variable
+
+#### DigitalOcean
+
+* Update Fedora Atomic module to use Fedora Atomic 28 ([#225](https://github.com/poseidon/typhoon/pull/225))
+  * Fedora Atomic 27 images disappeared from DigitalOcean and forced this early update
+
+#### Addons
+
+* Fix Prometheus data directory location ([#203](https://github.com/poseidon/typhoon/pull/203))
+* Configure Prometheus to scrape Kubelets directly with bearer token auth instead of proxying through the apiserver ([#217](https://github.com/poseidon/typhoon/pull/217))
+  * Security improvement: Drop RBAC permission from `nodes/proxy` to `nodes/metrics`
+  * Scale: Remove per-node proxied scrape load from the apiserver
+* Update Grafana from v5.04 to v5.1.3 ([#208](https://github.com/poseidon/typhoon/pull/208))
+  * Disable Grafana Google Analytics by default ([#214](https://github.com/poseidon/typhoon/issues/214))
+* Update nginx-ingress from 0.14.0 to 0.15.0
+* Annotate nginx-ingress service so Prometheus auto-discovers and scrapes service endpoints ([#222](https://github.com/poseidon/typhoon/pull/222))
+
+## v1.10.2
+
+* Kubernetes [v1.10.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1102)
+* [Introduce](https://typhoon.psdn.io/announce/#april-26-2018) Typhoon for Fedora Atomic ([#199](https://github.com/poseidon/typhoon/pull/199))
+* Update Calico from v3.0.4 to v3.1.1 ([#197](https://github.com/poseidon/typhoon/pull/197))
+  * https://www.projectcalico.org/announcing-calico-v3-1/
+  * https://github.com/projectcalico/calico/releases/tag/v3.1.0
+* Update etcd from v3.3.3 to v3.3.4
+* Update kube-dns from v1.14.9 to v1.14.10
+
+#### Google Cloud
+
+* Add support for multi-controller clusters (i.e. multi-master) ([#54](https://github.com/poseidon/typhoon/issues/54), [#190](https://github.com/poseidon/typhoon/pull/190))
+  * Switch from Google Cloud network load balancer to a TCP proxy load balancer. Avoid a [bug](https://issuetracker.google.com/issues/67366622) in Google network load balancers that limited clusters to only bootstrapping one controller node. 
+  * Add TCP health check for apiserver pods on controllers. Replace kubelet check approximation.
+
+#### Addons
+
+* Update nginx-ingress from 0.12.0 to 0.14.0
+* Update kube-state-metrics from v1.3.0 to v1.3.1
+
+## v1.10.1
+
+* Kubernetes [v1.10.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1101)
+* Enable etcd v3.3 metrics endpoint ([#175](https://github.com/poseidon/typhoon/pull/175))
+* Use `k8s.gcr.io` instead of `gcr.io/google_containers` ([#180](https://github.com/poseidon/typhoon/pull/180))
+  * Kubernetes [recommends](https://groups.google.com/forum/#!msg/kubernetes-dev/ytjk_rNrTa0/3EFUHvovCAAJ) using the alias to pull from the nearest regional mirror and to abstract the backing container registry
+* Update etcd from v3.3.2 to v3.3.3
+* Update kube-dns from v1.14.8 to v1.14.9
+* Use kubernetes-incubator/bootkube v0.12.0
+
+#### Bare-Metal
+
+* Fix need for multiple `terraform apply` runs to create a cluster with Terraform v0.11.4 ([#181](https://github.com/poseidon/typhoon/pull/181))
+  * To SSH during a disk install for debugging, SSH as user "core" with port 2222
+  * Remove the old trick of using a user "debug" during disk install
+
+#### Google Cloud
+
+* Refactor out the `controller` internal module
+
+#### Addons
+
+* Add Prometheus discovery for etcd peers on controller nodes ([#175](https://github.com/poseidon/typhoon/pull/175))
+  * Scrape etcd v3.3 `--listen-metrics-urls` for metrics
+  * Enable etcd alerts and populate the etcd Grafana dashboard
+* Update kube-state-metrics from v1.2.0 to v1.3.0
+
+## v1.10.0
+
+* Kubernetes [v1.10.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1100)
+* Remove unused, unmaintained `pxe-worker` internal module
+
+#### AWS
+
+* Add `disk_type` optional variable for setting the EBS volume type ([#176](https://github.com/poseidon/typhoon/pull/176))
+  * Change default type from `standard` to `gp2`. Prometheus etcd alerts are tuned for fast disks.
+
+#### Digital Ocean
+
+* Ensure etcd secrets are only distributed to controller hosts, not workers.
+* Remove `networking` optional variable. Only flannel works on Digital Ocean.
+
+#### Google Cloud
+
+* Add `disk_size` optional variable for setting instance disk size in GB
+* Add `controller_type` optional variable for setting machine type for controllers
+* Add `worker_type` optional variable for setting machine type for workers
+* Remove `machine_type` optional variable. Use `controller_type` and `worker_type`.
+
+#### Addons
+
+* Update Grafana from v4.6.3 to v5.0.4 ([#153](https://github.com/poseidon/typhoon/pull/153), [#174](https://github.com/poseidon/typhoon/pull/174))
+  * Restrict dashboard organization role to Viewer
+
+## v1.9.6
+
+* Kubernetes [v1.9.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v196)
+* Update Calico from v3.0.3 to v3.0.4
+
+#### Addons
+
+* Update heapster from v1.5.1 to v1.5.2
+
+## v1.9.5
+
+* Kubernetes [v1.9.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v195)
+  * Fix `subPath` volume mounts regression ([kubernetes#61076](https://github.com/kubernetes/kubernetes/issues/61076))
+* Introduce [Container Linux Config snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) on cloud platforms ([#145](https://github.com/poseidon/typhoon/pull/145))
+  * Validate and additively merge custom Container Linux Configs during `terraform plan`
+  * Define files, systemd units, dropins, networkd configs, mounts, users, and more
+  * Require updating `terraform-provider-ct` plugin from v0.2.0 to v0.2.1
+* Add `node-role.kubernetes.io/controller="true"` node label to controllers ([#160](https://github.com/poseidon/typhoon/pull/160))
+
+#### AWS
+
+* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
+
+#### Digital Ocean
+
+* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
+
+#### Google Cloud
+
+* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
+* Relax `os_image` to optional. Default to "coreos-stable".
+
+#### Addons
+
+* Update nginx-ingress from 0.11.0 to 0.12.0
+* Update Prometheus from 2.2.0 to 2.2.1
+
 ## v1.9.4
 
 * Kubernetes [v1.9.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v194)
@@ -15,7 +328,7 @@ Notable changes between versions.
 * Allow flexvolume plugins to be used on any Typhoon cluster (not just bare-metal)
 * Upgrade etcd from v3.2.15 to v3.3.2
 * Update Calico from v3.0.2 to v3.0.3
-* Use kubernetes-incubator/bootkube v0.10.0
+* Use kubernetes-incubator/bootkube v0.11.0
 * [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action recommended)
 
 #### AWS

README.md

@@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.9.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/google-cloud/#preemption) (varies by platform)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/cl/google-cloud/#preemption) (varies by platform)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Modules
@@ -24,61 +24,62 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
+| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
+| Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | beta |
+| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
+| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
 
-## Usage
+The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
+
+## Documentation
 
 * [Docs](https://typhoon.psdn.io)
-* [Concepts](https://typhoon.psdn.io/concepts/)
+* Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
-* Tutorials
+* Tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [Digital Ocean](cl/digital-ocean.md), and [Google-Cloud](cl/google-cloud.md)
-  * [AWS](https://typhoon.psdn.io/aws/)
-  * [Bare-Metal](https://typhoon.psdn.io/bare-metal/)
-  * [Digital Ocean](https://typhoon.psdn.io/digital-ocean/)
-  * [Google-Cloud](https://typhoon.psdn.io/google-cloud/)
 
-## Example
+## Usage
 
 Define a Kubernetes cluster by using the Terraform module for your chosen platform and operating system. Here's a minimal example:
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.9.4"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.11.3"
   
   providers = {
-    google = "google.default"
+    google   = "google.default"
-    local = "local.default"
+    local    = "local.default"
-    null = "null.default"
+    null     = "null.default"
     template = "template.default"
-    tls = "tls.default"
+    tls      = "tls.default"
   }
 
   # Google Cloud
+  cluster_name  = "yavin"
   region        = "us-central1"
   dns_zone      = "example.com"
   dns_zone_name = "example-zone"
-  os_image      = "coreos-stable"
 
-  cluster_name       = "yavin"
+  # configuration
-  controller_count   = 1
-  worker_count       = 2
   ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-
+  asset_dir          = "/home/user/.secrets/clusters/yavin"
-  # output assets dir
+  
-  asset_dir = "/home/user/.secrets/clusters/yavin"
+  # optional
+  worker_count = 2
 }
 ```
 
-Fetch modules, plan the changes to be made, and apply the changes.
+Initialize modules, plan the changes to be made, and apply the changes.
 
 ```sh
 $ terraform init
-$ terraform get --update
 $ terraform plan
-Plan: 37 to add, 0 to change, 0 to destroy.
+Plan: 64 to add, 0 to change, 0 to destroy.
 $ terraform apply
-Apply complete! Resources: 37 added, 0 changed, 0 destroyed.
+Apply complete! Resources: 64 added, 0 changed, 0 destroyed.
 ```
 
 In 4-8 minutes (varies by platform), the cluster will be ready. This Google Cloud example creates a `yavin.example.com` DNS record to resolve to a network load balancer across controller nodes.
@@ -87,9 +88,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.9.4
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.11.3
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.9.4
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.11.3
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.9.4
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.11.3
 ```
 
 List the pods.
@@ -100,10 +101,10 @@ NAMESPACE     NAME                                      READY  STATUS    RESTART
 kube-system   calico-node-1cs8z                         2/2    Running   0         6m
 kube-system   calico-node-d1l5b                         2/2    Running   0         6m
 kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
 kube-system   kube-apiserver-zppls                      1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-gh9kt  1/1    Running   0         6m
 kube-system   kube-controller-manager-3271970485-h90v8  1/1    Running   1         6m
-kube-system   kube-dns-1187388186-zj5dl                 3/3    Running   0         6m
 kube-system   kube-proxy-117v6                          1/1    Running   0         6m
 kube-system   kube-proxy-9886n                          1/1    Running   0         6m
 kube-system   kube-proxy-njn47                          1/1    Running   0         6m

addons/cluo/update-agent.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
       - name: update-agent
-        image: quay.io/coreos/container-linux-update-operator:v0.6.0
+        image: quay.io/coreos/container-linux-update-operator:v0.7.0
         command:
         - "/bin/update-agent"
         volumeMounts:

addons/cluo/update-operator.yaml

@@ -15,7 +15,7 @@ spec:
     spec:
       containers:
       - name: update-operator
-        image: quay.io/coreos/container-linux-update-operator:v0.6.0
+        image: quay.io/coreos/container-linux-update-operator:v0.7.0
         command:
         - "/bin/update-operator"
         env:

addons/grafana/dashboard-providers.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-providers
+  namespace: monitoring
+data:
+  dashboard-providers.yaml: |+
+    apiVersion: 1
+    providers:
+    - name: 'default'
+      ordId: 1
+      folder: ''
+      type: file
+      options:
+        path: /var/lib/grafana/dashboards

addons/grafana/dashboards.yaml

@@ -5,14 +5,12 @@ metadata:
   namespace: monitoring
 data:
   deployment-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -39,7 +37,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -110,7 +108,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -181,7 +179,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "Bps",
               "gauge": {
@@ -262,7 +260,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -333,7 +331,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -403,7 +401,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -473,7 +471,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -550,7 +548,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -665,7 +663,7 @@ data:
           {
             "allValue": ".*",
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": false,
             "label": "Namespace",
@@ -685,7 +683,7 @@ data:
           {
             "allValue": null,
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": false,
             "label": "Deployment",
@@ -737,24 +735,11 @@ data:
       "title": "Deployment",
       "version": 1
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   etcd-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "label": "prometheus",
           "description": "",
           "type": "datasource",
@@ -813,7 +798,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "format": "none",
@@ -889,7 +874,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 0,
@@ -978,7 +963,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 0,
@@ -1079,7 +1064,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "decimals": null,
               "editable": false,
               "error": false,
@@ -1161,7 +1146,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 0,
@@ -1250,7 +1235,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 0,
@@ -1342,7 +1327,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 5,
@@ -1422,7 +1407,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 5,
@@ -1502,7 +1487,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 0,
@@ -1582,7 +1567,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "decimals": null,
               "editable": false,
               "error": false,
@@ -1676,7 +1661,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 0,
@@ -1782,7 +1767,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "decimals": 0,
               "editable": false,
               "error": false,
@@ -1909,26 +1894,13 @@ data:
       "title": "etcd",
       "version": 4
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   kubernetes-capacity-planning-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -1954,7 +1926,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2032,7 +2004,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2134,7 +2106,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2250,7 +2222,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -2333,7 +2305,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2440,7 +2412,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percentunit",
               "gauge": {
@@ -2522,7 +2494,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2604,7 +2576,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2695,7 +2667,7 @@ data:
               "aliasColors": {},
               "bars": false,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -2782,7 +2754,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -2897,26 +2869,13 @@ data:
       "title": "Kubernetes Capacity Planning",
       "version": 4
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   kubernetes-cluster-health-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -2944,7 +2903,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3025,7 +2984,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3101,7 +3060,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3177,7 +3136,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3263,7 +3222,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3339,7 +3298,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3415,7 +3374,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3491,7 +3450,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3605,26 +3564,13 @@ data:
       "title": "Kubernetes Cluster Health",
       "version": 9
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   kubernetes-cluster-status-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -3651,7 +3597,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3723,7 +3669,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -3805,7 +3751,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -3877,7 +3823,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -3949,7 +3895,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4021,7 +3967,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -4103,7 +4049,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4175,7 +4121,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4247,7 +4193,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4319,7 +4265,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4429,26 +4375,13 @@ data:
       "title": "Kubernetes Cluster Status",
       "version": 3
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   kubernetes-control-plane-status-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -4475,7 +4408,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4550,7 +4483,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4625,7 +4558,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4700,7 +4633,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -4783,7 +4716,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -4869,7 +4802,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -4944,7 +4877,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -5069,26 +5002,13 @@ data:
       "title": "Kubernetes Control Plane Status",
       "version": 3
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   kubernetes-resource-requests-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -5113,7 +5033,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "description": "This represents the total [CPU resource requests](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) in the cluster.\nFor comparison the total [allocatable CPU cores](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md) is also shown.",
               "editable": false,
               "error": false,
@@ -5202,7 +5122,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -5284,7 +5204,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "description": "This represents the total [memory resource requests](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) in the cluster.\nFor comparison the total [allocatable memory](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md) is also shown.",
               "editable": false,
               "error": false,
@@ -5373,7 +5293,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -5486,26 +5406,13 @@ data:
       "title": "Kubernetes Resource Requests",
       "version": 2
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   nodes-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -5532,7 +5439,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -5611,7 +5518,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -5713,7 +5620,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -5825,7 +5732,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percent",
               "gauge": {
@@ -5907,7 +5814,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -6014,7 +5921,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(245, 54, 54, 0.9)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "percentunit",
               "gauge": {
@@ -6096,7 +6003,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -6178,7 +6085,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -6270,7 +6177,7 @@ data:
           {
             "allValue": null,
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": false,
             "label": null,
@@ -6322,26 +6229,13 @@ data:
       "title": "Nodes",
       "version": 2
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   pods-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -6366,7 +6260,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -6472,7 +6366,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -6576,7 +6470,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -6662,7 +6556,7 @@ data:
           {
             "allValue": ".*",
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": true,
             "label": "Namespace",
@@ -6682,7 +6576,7 @@ data:
           {
             "allValue": null,
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": false,
             "label": "Pod",
@@ -6702,7 +6596,7 @@ data:
           {
             "allValue": ".*",
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": true,
             "label": "Container",
@@ -6754,26 +6648,13 @@ data:
       "title": "Pods",
       "version": 1
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
   statefulset-dashboard.json: |+
-    {
-      "dashboard":
     {
       "__inputs": [
         {
           "description": "",
           "label": "prometheus",
-          "name": "DS_PROMETHEUS",
+          "name": "prometheus",
           "pluginId": "prometheus",
           "pluginName": "Prometheus",
           "type": "datasource"
@@ -6800,7 +6681,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -6871,7 +6752,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -6942,7 +6823,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "Bps",
               "gauge": {
@@ -7023,7 +6904,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -7094,7 +6975,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -7164,7 +7045,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -7234,7 +7115,7 @@ data:
                 "rgba(237, 129, 40, 0.89)",
                 "rgba(50, 172, 45, 0.97)"
               ],
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "format": "none",
               "gauge": {
@@ -7311,7 +7192,7 @@ data:
               "bars": false,
               "dashLength": 10,
               "dashes": false,
-              "datasource": "${DS_PROMETHEUS}",
+              "datasource": "prometheus",
               "editable": false,
               "error": false,
               "fill": 1,
@@ -7405,7 +7286,7 @@ data:
           {
             "allValue": ".*",
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": false,
             "label": "Namespace",
@@ -7425,7 +7306,7 @@ data:
           {
             "allValue": null,
             "current": {},
-            "datasource": "${DS_PROMETHEUS}",
+            "datasource": "prometheus",
             "hide": 0,
             "includeAll": false,
             "label": "StatefulSet",
@@ -7477,23 +7358,4 @@ data:
       "title": "StatefulSet",
       "version": 1
     }
-    ,
-      "inputs": [
-        {
-          "name": "DS_PROMETHEUS",
-          "pluginId": "prometheus",
-          "type": "datasource",
-          "value": "prometheus"
-        }
-      ],
-      "overwrite": true
-    }
-  prometheus-datasource.json: |+
-    {
-        "access": "proxy",
-        "basicAuth": false,
-        "name": "prometheus",
-        "type": "prometheus",
-        "url": "http://prometheus.monitoring.svc"
-    }
 ---

addons/grafana/datasources.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-datasources
+  namespace: monitoring
+data:
+  prometheus.yaml: |+
+    apiVersion: 1
+    datasources:
+    - name: prometheus
+      type: prometheus
+      access: proxy
+      orgId: 1
+      url: http://prometheus.monitoring.svc.cluster.local
+      version: 1
+      editable: false

addons/grafana/deployment.yaml

@@ -21,7 +21,7 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:4.6.3
+          image: grafana/grafana:5.2.4
           env:
             - name: GF_SERVER_HTTP_PORT
               value: "8080"
@@ -30,7 +30,9 @@ spec:
             - name: GF_AUTH_ANONYMOUS_ENABLED
               value: "true"
             - name: GF_AUTH_ANONYMOUS_ORG_ROLE
-              value: Admin
+              value: Viewer
+            - name: GF_ANALYTICS_REPORTING_ENABLED
+              value: "false"
           ports:
             - name: http
               containerPort: 8080
@@ -41,22 +43,20 @@ spec:
             limits:
               memory: 200Mi
               cpu: 200m
-        - name: grafana-watcher
-          image: quay.io/coreos/grafana-watcher:v0.0.8
-          args:
-            - '--watch-dir=/etc/grafana/dashboards'
-            - '--grafana-url=http://localhost:8080'
-          resources:
-            requests:
-              memory: "16Mi"
-              cpu: "50m"
-            limits:
-              memory: "32Mi"
-              cpu: "100m"
           volumeMounts:
-          - name: dashboards
+            - name: datasources
-            mountPath: /etc/grafana/dashboards
+              mountPath: /etc/grafana/provisioning/datasources
+            - name: dashboard-providers
+              mountPath: /etc/grafana/provisioning/dashboards
+            - name: dashboards
+              mountPath: /var/lib/grafana/dashboards
       volumes:
+        - name: datasources
+          configMap:
+            name: grafana-datasources
+        - name: dashboard-providers
+          configMap:
+            name: grafana-dashboard-providers
         - name: dashboards
           configMap:
             name: grafana-dashboards

addons/grafana/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: grafana
   namespace: monitoring
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '8080'
 spec:
   type: ClusterIP
   selector:

addons/heapster/deployment.yaml

@@ -14,11 +14,13 @@ spec:
       labels:
         name: heapster
         phase: prod
+      annotations:
+        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
     spec:
       serviceAccountName: heapster
       containers:
         - name: heapster
-          image: k8s.gcr.io/heapster-amd64:v1.5.1
+          image: k8s.gcr.io/heapster-amd64:v1.5.4
           command:
             - /heapster
             - --source=kubernetes.summary_api:''

addons/nginx-ingress/aws/0-namespace.yaml

@@ -2,3 +2,5 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ingress
+  labels:
+    name: ingress

addons/nginx-ingress/aws/default-backend/deployment.yaml

@@ -20,7 +20,7 @@ spec:
           # Any image is permissable as long as:
           # 1. It serves a 404 page at /
           # 2. It serves 200 on a /healthz endpoint
-          image: gcr.io/google_containers/defaultbackend:1.4
+          image: k8s.gcr.io/defaultbackend:1.4
           ports:
             - containerPort: 8080
           resources:

addons/nginx-ingress/aws/deployment.yaml

@@ -20,10 +20,9 @@ spec:
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
-      hostNetwork: true
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend
@@ -67,5 +66,12 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
+          securityContext:
+            capabilities:
+              add:
+              - NET_BIND_SERVICE
+              drop:
+              - ALL
+            runAsUser: 33 # www-data
       restartPolicy: Always
       terminationGracePeriodSeconds: 60

addons/nginx-ingress/aws/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: nginx-ingress-controller
   namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
 spec:
   type: ClusterIP
   selector:

addons/nginx-ingress/azure/0-namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ingress
+  labels:
+    name: ingress

addons/nginx-ingress/azure/default-backend/deployment.yaml

@@ -0,0 +1,40 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: default-backend
+  namespace: ingress
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: default-backend
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: default-backend
+        phase: prod
+    spec:
+      containers:
+        - name: default-backend
+          # Any image is permissable as long as:
+          # 1. It serves a 404 page at /
+          # 2. It serves 200 on a /healthz endpoint
+          image: k8s.gcr.io/defaultbackend:1.4
+          ports:
+            - containerPort: 8080
+          resources:
+            limits:
+              cpu: 10m
+              memory: 20Mi
+            requests:
+              cpu: 10m
+              memory: 20Mi
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+              scheme: HTTP
+            initialDelaySeconds: 30
+            timeoutSeconds: 5
+      terminationGracePeriodSeconds: 60

addons/nginx-ingress/azure/default-backend/service.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: default-backend
+  namespace: ingress
+spec:
+  type: ClusterIP
+  selector:
+    name: default-backend
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 8080

addons/nginx-ingress/azure/deployment.yaml

@@ -0,0 +1,77 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nginx-ingress-controller
+  namespace: ingress
+spec:
+  replicas: 2
+  strategy:
+    rollingUpdate:
+      maxUnavailable: 1
+  selector:
+    matchLabels:
+      name: nginx-ingress-controller
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: nginx-ingress-controller
+        phase: prod
+    spec:
+      nodeSelector:
+        node-role.kubernetes.io/node: ""
+      containers:
+        - name: nginx-ingress-controller
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          args:
+            - /nginx-ingress-controller
+            - --default-backend-service=$(POD_NAMESPACE)/default-backend
+            - --ingress-class=public
+          # use downward API
+          env:
+            - name: POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          ports:
+            - name: http
+              containerPort: 80
+              hostPort: 80
+            - name: https
+              containerPort: 443
+              hostPort: 443
+            - name: health
+              containerPort: 10254
+              hostPort: 10254
+          livenessProbe:
+            failureThreshold: 3
+            httpGet:
+              path: /healthz
+              port: 10254
+              scheme: HTTP
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            successThreshold: 1
+            timeoutSeconds: 1
+          readinessProbe:
+            failureThreshold: 3
+            httpGet:
+              path: /healthz
+              port: 10254
+              scheme: HTTP
+            periodSeconds: 10
+            successThreshold: 1
+            timeoutSeconds: 1
+          securityContext:
+            capabilities:
+              add:
+              - NET_BIND_SERVICE
+              drop:
+              - ALL
+            runAsUser: 33 # www-data
+      restartPolicy: Always
+      terminationGracePeriodSeconds: 60

addons/nginx-ingress/azure/rbac/cluster-role-binding.yaml

@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: ingress
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: ingress
+subjects:
+  - kind: ServiceAccount
+    namespace: ingress
+    name: default

addons/nginx-ingress/azure/rbac/cluster-role.yaml

@@ -0,0 +1,51 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: ingress
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - endpoints
+      - nodes
+      - pods
+      - secrets
+    verbs:
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - services
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+        - events
+    verbs:
+        - create
+        - patch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses/status
+    verbs:
+      - update

addons/nginx-ingress/azure/rbac/role-binding.yaml

@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ingress
+  namespace: ingress
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ingress
+subjects:
+  - kind: ServiceAccount
+    namespace: ingress
+    name: default

addons/nginx-ingress/azure/rbac/role.yaml

@@ -0,0 +1,41 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ingress
+  namespace: ingress
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - pods
+      - secrets
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    resourceNames:
+      # Defaults to "<election-id>-<ingress-class>"
+      # Here: "<ingress-controller-leader>-<nginx>"
+      # This has to be adapted if you change either parameter
+      # when launching the nginx-ingress-controller.
+      - "ingress-controller-leader-public"
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - endpoints
+    verbs:
+      - get
+      - create
+      - update

addons/nginx-ingress/azure/service.yaml

@@ -0,0 +1,22 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: nginx-ingress-controller
+  namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
+spec:
+  type: ClusterIP
+  selector:
+    name: nginx-ingress-controller
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 80
+    - name: https
+      protocol: TCP
+      port: 443
+      targetPort: 443

addons/nginx-ingress/bare-metal/0-namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ingress
+  labels:
+    name: ingress

addons/nginx-ingress/bare-metal/default-backend/deployment.yaml

@@ -0,0 +1,40 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: default-backend
+  namespace: ingress
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: default-backend
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: default-backend
+        phase: prod
+    spec:
+      containers:
+        - name: default-backend
+          # Any image is permissable as long as:
+          # 1. It serves a 404 page at /
+          # 2. It serves 200 on a /healthz endpoint
+          image: k8s.gcr.io/defaultbackend:1.4
+          ports:
+            - containerPort: 8080
+          resources:
+            limits:
+              cpu: 10m
+              memory: 20Mi
+            requests:
+              cpu: 10m
+              memory: 20Mi
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+              scheme: HTTP
+            initialDelaySeconds: 30
+            timeoutSeconds: 5
+      terminationGracePeriodSeconds: 60

addons/nginx-ingress/bare-metal/default-backend/service.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: default-backend
+  namespace: ingress
+spec:
+  type: ClusterIP
+  selector:
+    name: default-backend
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 8080

addons/nginx-ingress/bare-metal/deployment.yaml

@@ -0,0 +1,73 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ingress-controller-public
+  namespace: ingress
+spec:
+  replicas: 2
+  strategy:
+    rollingUpdate:
+      maxUnavailable: 1
+  selector:
+    matchLabels:
+      name: ingress-controller-public
+      phase: prod
+  template:
+    metadata:
+      labels:
+        name: ingress-controller-public
+        phase: prod
+    spec:
+      containers:
+        - name: nginx-ingress-controller
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
+          args:
+            - /nginx-ingress-controller
+            - --default-backend-service=$(POD_NAMESPACE)/default-backend
+            - --ingress-class=public
+          # use downward API
+          env:
+            - name: POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          ports:
+            - name: http
+              containerPort: 80
+            - name: https
+              containerPort: 443
+            - name: health
+              containerPort: 10254
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 10254
+              scheme: HTTP
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            successThreshold: 1
+            failureThreshold: 3
+            timeoutSeconds: 1
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: 10254
+              scheme: HTTP
+            periodSeconds: 10
+            successThreshold: 1
+            failureThreshold: 3
+            timeoutSeconds: 1
+          securityContext:
+            capabilities:
+              add:
+              - NET_BIND_SERVICE
+              drop:
+              - ALL
+            runAsUser: 33 # www-data
+      restartPolicy: Always
+      terminationGracePeriodSeconds: 60
+

addons/nginx-ingress/bare-metal/rbac/cluster-role-binding.yaml

@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: ingress
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: ingress
+subjects:
+  - kind: ServiceAccount
+    namespace: ingress
+    name: default

addons/nginx-ingress/bare-metal/rbac/cluster-role.yaml

@@ -0,0 +1,51 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: ingress
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - endpoints
+      - nodes
+      - pods
+      - secrets
+    verbs:
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - services
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+        - events
+    verbs:
+        - create
+        - patch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses/status
+    verbs:
+      - update

addons/nginx-ingress/bare-metal/rbac/role-binding.yaml

@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ingress
+  namespace: ingress
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ingress
+subjects:
+  - kind: ServiceAccount
+    namespace: ingress
+    name: default

addons/nginx-ingress/bare-metal/rbac/role.yaml

@@ -0,0 +1,41 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ingress
+  namespace: ingress
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - pods
+      - secrets
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    resourceNames:
+      # Defaults to "<election-id>-<ingress-class>"
+      # Here: "<ingress-controller-leader>-<nginx>"
+      # This has to be adapted if you change either parameter
+      # when launching the nginx-ingress-controller.
+      - "ingress-controller-leader-public"
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - endpoints
+    verbs:
+      - get
+      - create
+      - update

addons/nginx-ingress/bare-metal/service.yaml

@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: ingress-controller-public
+  namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
+spec:
+  type: ClusterIP
+  clusterIP: 10.3.0.12
+  selector:
+    name: ingress-controller-public
+    phase: prod
+  ports:
+    - name: http
+      protocol: TCP
+      port: 80
+      targetPort: 80
+    - name: https
+      protocol: TCP
+      port: 443
+      targetPort: 443

addons/nginx-ingress/digital-ocean/0-namespace.yaml

@@ -2,3 +2,5 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ingress
+  labels:
+    name: ingress

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -20,10 +20,9 @@ spec:
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
-      hostNetwork: true
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend
@@ -67,5 +66,12 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
+          securityContext:
+            capabilities:
+              add:
+              - NET_BIND_SERVICE
+              drop:
+              - ALL
+            runAsUser: 33 # www-data
       restartPolicy: Always
       terminationGracePeriodSeconds: 60

addons/nginx-ingress/digital-ocean/default-backend/deployment.yaml

@@ -20,7 +20,7 @@ spec:
           # Any image is permissable as long as:
           # 1. It serves a 404 page at /
           # 2. It serves 200 on a /healthz endpoint
-          image: gcr.io/google_containers/defaultbackend:1.4
+          image: k8s.gcr.io/defaultbackend:1.4
           ports:
             - containerPort: 8080
           resources:

addons/nginx-ingress/digital-ocean/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: nginx-ingress-controller
   namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
 spec:
   type: ClusterIP
   selector:

addons/nginx-ingress/google-cloud/0-namespace.yaml

@@ -2,3 +2,5 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ingress
+  labels:
+    name: ingress

addons/nginx-ingress/google-cloud/default-backend/deployment.yaml

@@ -20,7 +20,7 @@ spec:
           # Any image is permissable as long as:
           # 1. It serves a 404 page at /
           # 2. It serves 200 on a /healthz endpoint
-          image: gcr.io/google_containers/defaultbackend:1.4
+          image: k8s.gcr.io/defaultbackend:1.4
           ports:
             - containerPort: 8080
           resources:

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -20,10 +20,9 @@ spec:
     spec:
       nodeSelector:
         node-role.kubernetes.io/node: ""
-      hostNetwork: true
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend
@@ -67,5 +66,12 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
+          securityContext:
+            capabilities:
+              add:
+              - NET_BIND_SERVICE
+              drop:
+              - ALL
+            runAsUser: 33 # www-data
       restartPolicy: Always
       terminationGracePeriodSeconds: 60

addons/nginx-ingress/google-cloud/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: nginx-ingress-controller
   namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
 spec:
   type: ClusterIP
   selector:

addons/prometheus/0-namespace.yaml

@@ -2,3 +2,5 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: monitoring
+  labels:
+    name: monitoring

addons/prometheus/config.yaml

@@ -56,12 +56,7 @@ data:
         target_label: job
 
     # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
-    # metrics from a node by scraping kubelet (127.0.0.1:10255/metrics).
+    # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
-    #
-    # Rather than connecting directly to the node, the scrape is proxied though the
-    # Kubernetes apiserver.  This means it will work if Prometheus is running out of
-    # cluster, or can't connect to nodes for some other reason (e.g. because of
-    # firewalling).
     - job_name: 'kubelet'
       kubernetes_sd_configs:
       - role: node
@@ -69,48 +64,48 @@ data:
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        # Kubelet certs don't have any fixed IP SANs
+        insecure_skip_verify: true
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
-      - target_label: __address__
-        replacement: kubernetes.default.svc:443
-      - source_labels: [__meta_kubernetes_node_name]
-        regex: (.+)
-        target_label: __metrics_path__
-        replacement: /api/v1/nodes/${1}/proxy/metrics
 
     # Scrape config for Kubelet cAdvisor. Explore metrics from a node by
-    # scraping kubelet (127.0.0.1:10255/metrics/cadvisor).
+    # scraping kubelet (127.0.0.1:10250/metrics/cadvisor).
-    #
-    # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
-    # (those whose names begin with 'container_') have been removed from the
-    # Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
-    # retrieve those metrics.
-    #
-    # Rather than connecting directly to the node, the scrape is proxied though the
-    # Kubernetes apiserver.  This means it will work if Prometheus is running out of
-    # cluster, or can't connect to nodes for some other reason (e.g. because of
-    # firewalling).
     - job_name: 'kubernetes-cadvisor'
       kubernetes_sd_configs:
       - role: node
-      
+
       scheme: https
+      metrics_path: /metrics/cadvisor
       tls_config:
+        # Kubelet certs don't have any fixed IP SANs
+        insecure_skip_verify: true
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
-      - target_label: __address__
+
-        replacement: kubernetes.default.svc:443
+
-      - source_labels: [__meta_kubernetes_node_name]
+    # Scrap etcd metrics from controllers via listen-metrics-urls
-        regex: (.+)
+    - job_name: 'etcd'
-        target_label: __metrics_path__
+      kubernetes_sd_configs:
-        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
+      - role: node
+      scheme: http
+      relabel_configs:
+        - source_labels: [__meta_kubernetes_node_label_node_role_kubernetes_io_controller]
+          action: keep
+          regex: 'true'
+        - action: labelmap
+          regex: __meta_kubernetes_node_label_(.+)
+        - source_labels: [__meta_kubernetes_node_name]
+          action: replace
+          target_label: __address__
+          replacement: '${1}:2381'
     
     # Scrape config for service endpoints.
     #

addons/prometheus/deployment.yaml

@@ -17,29 +17,41 @@ spec:
     spec:
       serviceAccountName: prometheus
       containers:
-      - name: prometheus
+        - name: prometheus
-        image: quay.io/prometheus/prometheus:v2.2.0
+          image: quay.io/prometheus/prometheus:v2.3.2
-        args:
+          args:
-          - '--config.file=/etc/prometheus/prometheus.yaml'
+            - --web.listen-address=0.0.0.0:9090
-        ports:
+            - --config.file=/etc/prometheus/prometheus.yaml
-        - name: web
+            - --storage.tsdb.path=/var/lib/prometheus
-          containerPort: 9090
+          ports:
-        volumeMounts:
+            - name: web
-        - name: config
+              containerPort: 9090
-          mountPath: /etc/prometheus
+          volumeMounts:
-        - name: rules
+            - name: config
-          mountPath: /etc/prometheus/rules
+              mountPath: /etc/prometheus
-        - name: data
+            - name: rules
-          mountPath: /var/lib/prometheus
+              mountPath: /etc/prometheus/rules
-      dnsPolicy: ClusterFirst
+            - name: data
-      restartPolicy: Always
+              mountPath: /var/lib/prometheus
+          readinessProbe:
+            httpGet:
+              path: /-/ready
+              port: 9090
+            initialDelaySeconds: 10
+            timeoutSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /-/healthy
+              port: 9090
+            initialDelaySeconds: 10
+            timeoutSeconds: 10
       terminationGracePeriodSeconds: 30
       volumes:
-      - name: config
+        - name: config
-        configMap:
+          configMap:
-          name: prometheus-config
+            name: prometheus-config
-      - name: rules
+        - name: rules
-        configMap:
+          configMap:
-          name: prometheus-rules
+            name: prometheus-rules
-      - name: data
+        - name: data
-        emptyDir: {}
+          emptyDir: {}

addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml

@@ -5,6 +5,8 @@ metadata:
 rules:
 - apiGroups: [""]
   resources:
+  - configmaps
+  - secrets
   - nodes
   - pods
   - services

addons/prometheus/exporters/kube-state-metrics/deployment.yaml

@@ -22,7 +22,7 @@ spec:
       serviceAccountName: kube-state-metrics
       containers:
       - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.2.0
+        image: quay.io/coreos/kube-state-metrics:v1.4.0
         ports:
           - name: metrics
             containerPort: 8080
@@ -33,7 +33,7 @@ spec:
           initialDelaySeconds: 5
           timeoutSeconds: 5
       - name: addon-resizer
-        image: gcr.io/google_containers/addon-resizer:1.7
+        image: k8s.gcr.io/addon-resizer:1.7
         resources:
           limits:
             cpu: 100m

addons/prometheus/rbac/cluster-role.yaml

@@ -6,7 +6,7 @@ rules:
 - apiGroups: [""]
   resources:
   - nodes
-  - nodes/proxy
+  - nodes/metrics
   - services
   - endpoints
   - pods

addons/prometheus/rules.yaml

@@ -63,26 +63,6 @@ data:
           description: etcd instance {{ $labels.instance }} has seen {{ $value }} leader
             changes within the last hour
           summary: a high number of leader changes within the etcd cluster are happening
-      - alert: HighNumberOfFailedGRPCRequests
-        expr: sum(rate(grpc_server_handled_total{grpc_code!="OK",job="etcd"}[5m])) BY (grpc_service, grpc_method)
-          / sum(rate(grpc_server_handled_total{job="etcd"}[5m])) BY (grpc_service, grpc_method) > 0.01
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          description: '{{ $value }}% of requests for {{ $labels.grpc_method }} failed
-            on etcd instance {{ $labels.instance }}'
-          summary: a high number of gRPC requests are failing
-      - alert: HighNumberOfFailedGRPCRequests
-        expr: sum(rate(grpc_server_handled_total{grpc_code!="OK",job="etcd"}[5m])) BY (grpc_service, grpc_method)
-          / sum(rate(grpc_server_handled_total{job="etcd"}[5m])) BY (grpc_service, grpc_method) > 0.05
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          description: '{{ $value }}% of requests for {{ $labels.grpc_method }} failed
-            on etcd instance {{ $labels.instance }}'
-          summary: a high number of gRPC requests are failing
       - alert: GRPCRequestsSlow
         expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job="etcd",grpc_type="unary"}[5m])) by (grpc_service, grpc_method, le))
           > 0.15
@@ -319,7 +299,7 @@ data:
         labels:
           severity: warning
         annotations:
-          description: Pod {{$labels.namespaces}}/{{$labels.pod}} is was restarted {{$value}}
+          description: Pod {{$labels.namespaces}}/{{$labels.pod}} restarted {{$value}}
             times within the last hour
           summary: Pod is restarting frequently
   kubelet.rules.yaml: |
@@ -516,6 +496,13 @@ data:
         annotations:
           description: device {{$labels.device}} on node {{$labels.instance}} is running
             full within the next 2 hours (mounted at {{$labels.mountpoint}})
+      - alert: InactiveRAIDDisk
+        expr: node_md_disks - node_md_disks_active > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          description: '{{$value}} RAID disk(s) on node {{$labels.instance}} are inactive'
   prometheus.rules.yaml: |
     groups:
     - name: prometheus.rules

aws/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.9.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
@@ -19,5 +19,5 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/aws/).
+Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
 

aws/container-linux/kubernetes/ami.tf

@@ -1,3 +1,13 @@
+locals {
+  # Pick a CoreOS Container Linux derivative
+  # coreos-stable -> Container Linux AMI
+  # flatcar-stable -> Flatcar Linux AMI
+  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+
+  flavor  = "${element(split("-", var.os_image), 0)}"
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
 data "aws_ami" "coreos" {
   most_recent = true
   owners      = ["595879546273"]
@@ -14,6 +24,26 @@ data "aws_ami" "coreos" {
 
   filter {
     name   = "name"
-    values = ["CoreOS-${var.os_channel}-*"]
+    values = ["CoreOS-${local.channel}-*"]
+  }
+}
+
+data "aws_ami" "flatcar" {
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
   }
 }

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=c5fc93d95fe4993511656cdd6372afbd1307f08f"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,12 +7,13 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.2"
+            Environment="ETCD_IMAGE_TAG=v3.3.9"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
             Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
             Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
+            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
             Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
             Environment="ETCD_STRICT_RECONFIG_CHECK=true"
             Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
@@ -55,6 +56,8 @@ systemd:
           --mount volume=resolv,target=/etc/resolv.conf \
           --volume var-lib-cni,kind=host,source=/var/lib/cni \
           --mount volume=var-lib-cni,target=/var/lib/cni \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
           --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
@@ -66,12 +69,14 @@ systemd:
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
         ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
         ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
         ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
-          --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -81,6 +86,7 @@ systemd:
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
           --node-labels=node-role.kubernetes.io/master \
+          --node-labels=node-role.kubernetes.io/controller="true" \
           --pod-manifest-path=/etc/kubernetes/manifests \
           --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
           --volume-plugin-dir=/var/lib/kubelet/volumeplugins
@@ -115,8 +121,8 @@ storage:
       mode: 0644
       contents:
         inline: |
-          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
+          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.9.4
+          KUBELET_IMAGE_TAG=v1.11.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -137,7 +143,7 @@ storage:
           # Move experimental manifests
           [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
           BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
-          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.11.0}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
           BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
           exec /usr/bin/rkt run \
             --trust-keys-from-https \

aws/container-linux/kubernetes/controllers.tf

@@ -23,12 +23,12 @@ resource "aws_instance" "controllers" {
 
   instance_type = "${var.controller_type}"
 
-  ami       = "${data.aws_ami.coreos.image_id}"
+  ami       = "${local.ami_id}"
   user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
 
   # storage
   root_block_device {
-    volume_type = "standard"
+    volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
   }
 
@@ -54,23 +54,23 @@ data "template_file" "controller_config" {
     etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
 
     # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", formatlist("%s=https://%s:2380", null_resource.repeat.*.triggers.name, null_resource.repeat.*.triggers.domain))}"
+    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
 
-    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
-    ssh_authorized_key    = "${var.ssh_authorized_key}"
-    cluster_domain_suffix = "${var.cluster_domain_suffix}"
     kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
   }
 }
 
-# Horrible hack to generate a Terraform list of a desired length without dependencies.
+data "template_file" "etcds" {
-# Ideal ${repeat("etcd", 3) -> ["etcd", "etcd", "etcd"]}
+  count    = "${var.controller_count}"
-resource null_resource "repeat" {
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-  count = "${var.controller_count}"
 
-  triggers {
+  vars {
-    name   = "etcd${count.index}"
+    index        = "${count.index}"
-    domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+    cluster_name = "${var.cluster_name}"
+    dns_zone     = "${var.dns_zone}"
   }
 }
 
@@ -78,4 +78,5 @@ data "ct_config" "controller_ign" {
   count        = "${var.controller_count}"
   content      = "${element(data.template_file.controller_config.*.rendered, count.index)}"
   pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
 }

aws/container-linux/kubernetes/nlb.tf

@@ -1,21 +1,21 @@
-# kube-apiserver Network Load Balancer DNS Record
+# Network Load Balancer DNS Record
 resource "aws_route53_record" "apiserver" {
   zone_id = "${var.dns_zone_id}"
 
   name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
   type = "A"
 
-  # AWS recommends their special "alias" records for ELBs
+  # AWS recommends their special "alias" records for NLBs
   alias {
-    name                   = "${aws_lb.apiserver.dns_name}"
+    name                   = "${aws_lb.nlb.dns_name}"
-    zone_id                = "${aws_lb.apiserver.zone_id}"
+    zone_id                = "${aws_lb.nlb.zone_id}"
     evaluate_target_health = true
   }
 }
 
-# Network Load Balancer for apiservers
+# Network Load Balancer for apiservers and ingress
-resource "aws_lb" "apiserver" {
+resource "aws_lb" "nlb" {
-  name               = "${var.cluster_name}-apiserver"
+  name               = "${var.cluster_name}-nlb"
   load_balancer_type = "network"
   internal           = false
 
@@ -24,11 +24,11 @@ resource "aws_lb" "apiserver" {
   enable_cross_zone_load_balancing = true
 }
 
-# Forward HTTP traffic to controllers
+# Forward TCP apiserver traffic to controllers
 resource "aws_lb_listener" "apiserver-https" {
-  load_balancer_arn = "${aws_lb.apiserver.arn}"
+  load_balancer_arn = "${aws_lb.nlb.arn}"
   protocol          = "TCP"
-  port              = "443"
+  port              = "6443"
 
   default_action {
     type             = "forward"
@@ -36,6 +36,30 @@ resource "aws_lb_listener" "apiserver-https" {
   }
 }
 
+# Forward HTTP ingress traffic to workers
+resource "aws_lb_listener" "ingress-http" {
+  load_balancer_arn = "${aws_lb.nlb.arn}"
+  protocol          = "TCP"
+  port              = 80
+
+  default_action {
+    type             = "forward"
+    target_group_arn = "${module.workers.target_group_http}"
+  }
+}
+
+# Forward HTTPS ingress traffic to workers
+resource "aws_lb_listener" "ingress-https" {
+  load_balancer_arn = "${aws_lb.nlb.arn}"
+  protocol          = "TCP"
+  port              = 443
+
+  default_action {
+    type             = "forward"
+    target_group_arn = "${module.workers.target_group_https}"
+  }
+}
+
 # Target group of controllers
 resource "aws_lb_target_group" "controllers" {
   name        = "${var.cluster_name}-controllers"
@@ -43,12 +67,12 @@ resource "aws_lb_target_group" "controllers" {
   target_type = "instance"
 
   protocol = "TCP"
-  port     = 443
+  port     = 6443
 
-  # Kubelet HTTP health check
+  # TCP health check for apiserver
   health_check {
     protocol = "TCP"
-    port     = 443
+    port     = 6443
 
     # NLBs required to use same healthy and unhealthy thresholds
     healthy_threshold   = 3
@@ -65,5 +89,5 @@ resource "aws_lb_target_group_attachment" "controllers" {
 
   target_group_arn = "${aws_lb_target_group.controllers.arn}"
   target_id        = "${element(aws_instance.controllers.*.id, count.index)}"
-  port             = 443
+  port             = 6443
 }

aws/container-linux/kubernetes/outputs.tf

@@ -1,5 +1,7 @@
+# Outputs for Kubernetes Ingress
+
 output "ingress_dns_name" {
-  value       = "${module.workers.ingress_dns_name}"
+  value       = "${aws_lb.nlb.dns_name}"
   description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }
 
@@ -23,3 +25,15 @@ output "worker_security_groups" {
 output "kubeconfig" {
   value = "${module.bootkube.kubeconfig}"
 }
+
+# Outputs for custom load balancing
+
+output "worker_target_group_http" {
+  description = "ARN of a target group of workers for HTTP traffic"
+  value       = "${module.workers.target_group_http}"
+}
+
+output "worker_target_group_https" {
+  description = "ARN of a target group of workers for HTTPS traffic"
+  value       = "${module.workers.target_group_https}"
+}

aws/container-linux/kubernetes/require.tf

@@ -1,11 +1,11 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "aws" {
-  version = "~> 1.11"
+  version = "~> 1.13"
 }
 
 provider "local" {

aws/container-linux/kubernetes/security.tf

@@ -11,16 +11,6 @@ resource "aws_security_group" "controller" {
   tags = "${map("Name", "${var.cluster_name}-controller")}"
 }
 
-resource "aws_security_group_rule" "controller-icmp" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "icmp"
-  from_port   = 0
-  to_port     = 0
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "controller-ssh" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -31,16 +21,6 @@ resource "aws_security_group_rule" "controller-ssh" {
   cidr_blocks = ["0.0.0.0/0"]
 }
 
-resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 443
-  to_port     = 443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "controller-etcd" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -51,6 +31,27 @@ resource "aws_security_group_rule" "controller-etcd" {
   self      = true
 }
 
+# Allow Prometheus to scrape etcd metrics
+resource "aws_security_group_rule" "controller-etcd-metrics" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 2381
+  to_port                  = 2381
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-apiserver" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 6443
+  to_port     = 6443
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
 resource "aws_security_group_rule" "controller-flannel" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -71,6 +72,7 @@ resource "aws_security_group_rule" "controller-flannel-self" {
   self      = true
 }
 
+# Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "controller-node-exporter" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -81,6 +83,17 @@ resource "aws_security_group_rule" "controller-node-exporter" {
   source_security_group_id = "${aws_security_group.worker.id}"
 }
 
+# Allow apiserver to access kubelets for exec, log, port-forward
+resource "aws_security_group_rule" "controller-kubelet" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10250
+  to_port                  = 10250
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
 resource "aws_security_group_rule" "controller-kubelet-self" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -91,6 +104,7 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
   self      = true
 }
 
+# Allow heapster / metrics-server to scrape kubelet read-only
 resource "aws_security_group_rule" "controller-kubelet-read" {
   security_group_id = "${aws_security_group.controller.id}"
 
@@ -193,16 +207,6 @@ resource "aws_security_group" "worker" {
   tags = "${map("Name", "${var.cluster_name}-worker")}"
 }
 
-resource "aws_security_group_rule" "worker-icmp" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "icmp"
-  from_port   = 0
-  to_port     = 0
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
 resource "aws_security_group_rule" "worker-ssh" {
   security_group_id = "${aws_security_group.worker.id}"
 
@@ -253,6 +257,7 @@ resource "aws_security_group_rule" "worker-flannel-self" {
   self      = true
 }
 
+# Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "worker-node-exporter" {
   security_group_id = "${aws_security_group.worker.id}"
 
@@ -273,6 +278,7 @@ resource "aws_security_group_rule" "ingress-health" {
   cidr_blocks = ["0.0.0.0/0"]
 }
 
+# Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "worker-kubelet" {
   security_group_id = "${aws_security_group.worker.id}"
 
@@ -283,6 +289,7 @@ resource "aws_security_group_rule" "worker-kubelet" {
   source_security_group_id = "${aws_security_group.controller.id}"
 }
 
+# Allow Prometheus to scrape kubelet metrics
 resource "aws_security_group_rule" "worker-kubelet-self" {
   security_group_id = "${aws_security_group.worker.id}"
 
@@ -293,6 +300,7 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
   self      = true
 }
 
+# Allow heapster / metrics-server to scrape kubelet read-only
 resource "aws_security_group_rule" "worker-kubelet-read" {
   security_group_id = "${aws_security_group.worker.id}"
 

aws/container-linux/kubernetes/ssh.tf

@@ -1,5 +1,5 @@
-# Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
+# Secure copy etcd TLS assets to controllers.
-resource "null_resource" "copy-secrets" {
+resource "null_resource" "copy-controller-secrets" {
   count = "${var.controller_count}"
 
   connection {
@@ -9,11 +9,6 @@ resource "null_resource" "copy-secrets" {
     timeout = "15m"
   }
 
-  provisioner "file" {
-    content     = "${module.bootkube.kubeconfig}"
-    destination = "$HOME/kubeconfig"
-  }
-
   provisioner "file" {
     content     = "${module.bootkube.etcd_ca_cert}"
     destination = "$HOME/etcd-client-ca.crt"
@@ -61,7 +56,6 @@ resource "null_resource" "copy-secrets" {
       "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
       "sudo chown -R etcd:etcd /etc/ssl/etcd",
       "sudo chmod -R 500 /etc/ssl/etcd",
-      "sudo mv /home/core/kubeconfig /etc/kubernetes/kubeconfig",
     ]
   }
 }
@@ -69,7 +63,12 @@ resource "null_resource" "copy-secrets" {
 # Secure copy bootkube assets to ONE controller and start bootkube to perform
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
-  depends_on = ["module.bootkube", "null_resource.copy-secrets", "aws_route53_record.apiserver"]
+  depends_on = [
+    "module.bootkube",
+    "module.workers",
+    "aws_route53_record.apiserver",
+    "null_resource.copy-controller-secrets",
+  ]
 
   connection {
     type    = "ssh"
@@ -85,7 +84,7 @@ resource "null_resource" "bootkube-start" {
 
   provisioner "remote-exec" {
     inline = [
-      "sudo mv /home/core/assets /opt/bootkube",
+      "sudo mv $HOME/assets /opt/bootkube",
       "sudo systemctl start bootkube",
     ]
   }

aws/container-linux/kubernetes/variables.tf

@@ -1,51 +1,26 @@
 variable "cluster_name" {
   type        = "string"
-  description = "Cluster name"
+  description = "Unique cluster name (prepended to dns_zone)"
 }
 
+# AWS
+
 variable "dns_zone" {
   type        = "string"
-  description = "AWS DNS Zone (e.g. aws.dghubble.io)"
+  description = "AWS Route53 DNS Zone (e.g. aws.example.com)"
 }
 
 variable "dns_zone_id" {
   type        = "string"
-  description = "AWS DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
+  description = "AWS Route53 DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
 }
 
-variable "ssh_authorized_key" {
+# instances
-  type        = "string"
-  description = "SSH public key for user 'core'"
-}
-
-variable "os_channel" {
-  type        = "string"
-  default     = "stable"
-  description = "Container Linux AMI channel (stable, beta, alpha)"
-}
-
-variable "disk_size" {
-  type        = "string"
-  default     = "40"
-  description = "The size of the disk in Gigabytes"
-}
-
-variable "host_cidr" {
-  description = "CIDR IPv4 range to assign to EC2 nodes"
-  type        = "string"
-  default     = "10.0.0.0/16"
-}
 
 variable "controller_count" {
   type        = "string"
   default     = "1"
-  description = "Number of controllers"
+  description = "Number of controllers (i.e. masters)"
-}
-
-variable "controller_type" {
-  type        = "string"
-  default     = "t2.small"
-  description = "Controller EC2 instance type"
 }
 
 variable "worker_count" {
@@ -54,13 +29,60 @@ variable "worker_count" {
   description = "Number of workers"
 }
 
+variable "controller_type" {
+  type        = "string"
+  default     = "t2.small"
+  description = "EC2 instance type for controllers"
+}
+
 variable "worker_type" {
   type        = "string"
   default     = "t2.small"
-  description = "Worker EC2 instance type"
+  description = "EC2 instance type for workers"
 }
 
-# bootkube assets
+variable "os_image" {
+  type        = "string"
+  default     = "coreos-stable"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+}
+
+variable "disk_size" {
+  type        = "string"
+  default     = "40"
+  description = "Size of the EBS volume in GB"
+}
+
+variable "disk_type" {
+  type        = "string"
+  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
+}
+
+variable "worker_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
+variable "controller_clc_snippets" {
+  type        = "list"
+  description = "Controller Container Linux Config snippets"
+  default     = []
+}
+
+variable "worker_clc_snippets" {
+  type        = "list"
+  description = "Worker Container Linux Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+  type        = "string"
+  description = "SSH public key for user 'core'"
+}
 
 variable "asset_dir" {
   description = "Path to a directory where generated assets should be placed (contains secrets)"
@@ -79,6 +101,12 @@ variable "network_mtu" {
   default     = "1480"
 }
 
+variable "host_cidr" {
+  description = "CIDR IPv4 range to assign to EC2 nodes"
+  type        = "string"
+  default     = "10.0.0.0/16"
+}
+
 variable "pod_cidr" {
   description = "CIDR IPv4 range to assign Kubernetes pods"
   type        = "string"
@@ -88,7 +116,7 @@ variable "pod_cidr" {
 variable "service_cidr" {
   description = <<EOD
 CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
   type    = "string"
@@ -96,7 +124,7 @@ EOD
 }
 
 variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
   type        = "string"
   default     = "cluster.local"
 }

aws/container-linux/kubernetes/workers.tf

@@ -8,12 +8,14 @@ module "workers" {
   security_groups = ["${aws_security_group.worker.id}"]
   count           = "${var.worker_count}"
   instance_type   = "${var.worker_type}"
-  os_channel      = "${var.os_channel}"
+  os_image        = "${var.os_image}"
   disk_size       = "${var.disk_size}"
+  spot_price      = "${var.worker_price}"
 
   # configuration
   kubeconfig            = "${module.bootkube.kubeconfig}"
   ssh_authorized_key    = "${var.ssh_authorized_key}"
   service_cidr          = "${var.service_cidr}"
   cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  clc_snippets          = "${var.worker_clc_snippets}"
 }

aws/container-linux/kubernetes/workers/ami.tf

@@ -1,3 +1,13 @@
+locals {
+  # Pick a CoreOS Container Linux derivative
+  # coreos-stable -> Container Linux AMI
+  # flatcar-stable -> Flatcar Linux AMI
+  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+
+  flavor  = "${element(split("-", var.os_image), 0)}"
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
 data "aws_ami" "coreos" {
   most_recent = true
   owners      = ["595879546273"]
@@ -14,6 +24,26 @@ data "aws_ami" "coreos" {
 
   filter {
     name   = "name"
-    values = ["CoreOS-${var.os_channel}-*"]
+    values = ["CoreOS-${local.channel}-*"]
+  }
+}
+
+data "aws_ami" "flatcar" {
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
   }
 }

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -31,6 +31,8 @@ systemd:
           --mount volume=resolv,target=/etc/resolv.conf \
           --volume var-lib-cni,kind=host,source=/var/lib/cni \
           --mount volume=var-lib-cni,target=/var/lib/cni \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
           --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
@@ -39,15 +41,15 @@ systemd:
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
         ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
         ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
         ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
-          --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -89,8 +91,8 @@ storage:
       mode: 0644
       contents:
         inline: |
-          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
+          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.9.4
+          KUBELET_IMAGE_TAG=v1.11.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -108,7 +110,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://gcr.io/google_containers/hyperkube:v1.9.4 \
+            docker://k8s.gcr.io/hyperkube:v1.11.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/ingress.tf

@@ -1,39 +1,4 @@
-# Network Load Balancer for Ingress
+# Target groups of instances for use with load balancers
-resource "aws_lb" "ingress" {
-  name               = "${var.name}-ingress"
-  load_balancer_type = "network"
-  internal           = false
-
-  subnets = ["${var.subnet_ids}"]
-
-  enable_cross_zone_load_balancing = true
-}
-
-# Forward HTTP traffic to workers
-resource "aws_lb_listener" "ingress-http" {
-  load_balancer_arn = "${aws_lb.ingress.arn}"
-  protocol          = "TCP"
-  port              = 80
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${aws_lb_target_group.workers-http.arn}"
-  }
-}
-
-# Forward HTTPS traffic to workers
-resource "aws_lb_listener" "ingress-https" {
-  load_balancer_arn = "${aws_lb.ingress.arn}"
-  protocol          = "TCP"
-  port              = 443
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${aws_lb_target_group.workers-https.arn}"
-  }
-}
-
-# Network Load Balancer target groups of instances
 
 resource "aws_lb_target_group" "workers-http" {
   name        = "${var.name}-workers-http"
@@ -43,7 +8,7 @@ resource "aws_lb_target_group" "workers-http" {
   protocol = "TCP"
   port     = 80
 
-  # Ingress Controller HTTP health check
+  # HTTP health check for ingress
   health_check {
     protocol = "HTTP"
     port     = 10254
@@ -66,7 +31,7 @@ resource "aws_lb_target_group" "workers-https" {
   protocol = "TCP"
   port     = 443
 
-  # Ingress Controller HTTP health check
+  # HTTP health check for ingress
   health_check {
     protocol = "HTTP"
     port     = 10254

aws/container-linux/kubernetes/workers/outputs.tf

@@ -1,4 +1,9 @@
-output "ingress_dns_name" {
+output "target_group_http" {
-  value       = "${aws_lb.ingress.dns_name}"
+  description = "ARN of a target group of workers for HTTP traffic"
-  description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
+  value       = "${aws_lb_target_group.workers-http.arn}"
+}
+
+output "target_group_https" {
+  description = "ARN of a target group of workers for HTTPS traffic"
+  value       = "${aws_lb_target_group.workers-https.arn}"
 }

aws/container-linux/kubernetes/workers/variables.tf

@@ -1,21 +1,23 @@
 variable "name" {
   type        = "string"
-  description = "Unique name instance group"
+  description = "Unique name for the worker pool"
 }
 
+# AWS
+
 variable "vpc_id" {
   type        = "string"
-  description = "ID of the VPC for creating instances"
+  description = "Must be set to `vpc_id` output by cluster"
 }
 
 variable "subnet_ids" {
   type        = "list"
-  description = "List of subnet IDs for creating instances"
+  description = "Must be set to `subnet_ids` output by cluster"
 }
 
 variable "security_groups" {
   type        = "list"
-  description = "List of security group IDs"
+  description = "Must be set to `worker_security_groups` output by cluster"
 }
 
 # instances
@@ -32,23 +34,41 @@ variable "instance_type" {
   description = "EC2 instance type"
 }
 
-variable "os_channel" {
+variable "os_image" {
   type        = "string"
-  default     = "stable"
+  default     = "coreos-stable"
-  description = "Container Linux AMI channel (stable, beta, alpha)"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
 }
 
 variable "disk_size" {
   type        = "string"
   default     = "40"
-  description = "Size of the disk in GB"
+  description = "Size of the EBS volume in GB"
+}
+
+variable "disk_type" {
+  type        = "string"
+  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
+}
+
+variable "spot_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
+variable "clc_snippets" {
+  type        = "list"
+  description = "Container Linux Config snippets"
+  default     = []
 }
 
 # configuration
 
 variable "kubeconfig" {
   type        = "string"
-  description = "Generated Kubelet kubeconfig"
+  description = "Must be set to `kubeconfig` output by cluster"
 }
 
 variable "ssh_authorized_key" {
@@ -59,7 +79,7 @@ variable "ssh_authorized_key" {
 variable "service_cidr" {
   description = <<EOD
 CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for kube-dns.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD
 
   type    = "string"
@@ -67,7 +87,7 @@ EOD
 }
 
 variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by kube-dns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
   type        = "string"
   default     = "cluster.local"
 }

aws/container-linux/kubernetes/workers/workers.tf

@@ -26,6 +26,12 @@ resource "aws_autoscaling_group" "workers" {
     create_before_destroy = true
   }
 
+  # Waiting for instance creation delays adding the ASG to state. If instances
+  # can't be created (e.g. spot price too low), the ASG will be orphaned.
+  # Orphaned ASGs escape cleanup, can't be updated, and keep bidding if spot is
+  # used. Disable wait to avoid issues and align with other clouds.
+  wait_for_capacity_timeout = "0"
+
   tags = [{
     key                 = "Name"
     value               = "${var.name}-worker"
@@ -35,14 +41,16 @@ resource "aws_autoscaling_group" "workers" {
 
 # Worker template
 resource "aws_launch_configuration" "worker" {
-  image_id      = "${data.aws_ami.coreos.image_id}"
+  image_id          = "${local.ami_id}"
-  instance_type = "${var.instance_type}"
+  instance_type     = "${var.instance_type}"
+  spot_price        = "${var.spot_price}"
+  enable_monitoring = false
 
   user_data = "${data.ct_config.worker_ign.rendered}"
 
   # storage
   root_block_device {
-    volume_type = "standard"
+    volume_type = "${var.disk_type}"
     volume_size = "${var.disk_size}"
   }
 
@@ -71,4 +79,5 @@ data "template_file" "worker_config" {
 data "ct_config" "worker_ign" {
   content      = "${data.template_file.worker_config.rendered}"
   pretty_print = false
+  snippets     = ["${var.clc_snippets}"]
 }

aws/fedora-atomic/kubernetes/LICENSE

@@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Typhoon Authors
+Copyright (c) 2017 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

aws/fedora-atomic/kubernetes/README.md

@@ -0,0 +1,23 @@
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
+
+* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
+

aws/fedora-atomic/kubernetes/ami.tf

@@ -0,0 +1,19 @@
+data "aws_ami" "fedora" {
+  most_recent = true
+  owners      = ["125523088429"]
+
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Fedora-AtomicHost-28-20180625.1.x86_64-*-gp2-*"]
+  }
+}

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -0,0 +1,17 @@
+# Self-hosted Kubernetes assets (kubeconfig, manifests)
+module "bootkube" {
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  etcd_servers          = ["${aws_route53_record.etcds.*.fqdn}"]
+  asset_dir             = "${var.asset_dir}"
+  networking            = "${var.networking}"
+  network_mtu           = "${var.network_mtu}"
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+
+  # Fedora
+  trusted_certs_dir = "/etc/pki/tls/certs"
+}

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -0,0 +1,108 @@
+#cloud-config
+write_files:
+  - path: /etc/etcd/etcd.conf
+    content: |
+      ETCD_NAME=${etcd_name}
+      ETCD_DATA_DIR=/var/lib/etcd
+      ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+      ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+      ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+      ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+      ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+      ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+      ETCD_STRICT_RECONFIG_CHECK=true
+      ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+      ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+      ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+      ETCD_CLIENT_CERT_AUTH=true
+      ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+      ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+      ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+      ETCD_PEER_CLIENT_CERT_AUTH=true
+  - path: /etc/systemd/system/cloud-metadata.service
+    content: |
+      [Unit]
+      Description=Cloud metadata agent
+      [Service]
+      Type=oneshot
+      Environment=OUTPUT=/run/metadata/cloud
+      ExecStart=/usr/bin/mkdir -p /run/metadata
+      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
+        --url http://169.254.169.254/latest/meta-data/local-ipv4\
+        --retry 10)" > $${OUTPUT}'
+      [Install]
+      WantedBy=multi-user.target
+  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
+    content: |
+      [Unit]
+      Requires=cloud-metadata.service
+      After=cloud-metadata.service
+      Wants=rpc-statd.service
+      [Service]
+      ExecStartPre=/bin/mkdir -p /opt/cni/bin
+      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+      ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
+      ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
+      ExecStartPre=/bin/mkdir -p /var/lib/cni
+      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+      Restart=always
+      RestartSec=10
+  - path: /etc/kubernetes/kubelet.conf
+    content: |
+      ARGS="--anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
+        --client-ca-file=/etc/kubernetes/ca.crt \
+        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_domain=${cluster_domain_suffix} \
+        --cni-conf-dir=/etc/kubernetes/cni/net.d \
+        --exit-on-lock-contention \
+        --kubeconfig=/etc/kubernetes/kubeconfig \
+        --lock-file=/var/run/lock/kubelet.lock \
+        --network-plugin=cni \
+        --node-labels=node-role.kubernetes.io/master \
+        --node-labels=node-role.kubernetes.io/controller="true" \
+        --pod-manifest-path=/etc/kubernetes/manifests \
+        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
+  - path: /etc/kubernetes/kubeconfig
+    permissions: '0644'
+    content: |
+      ${kubeconfig}
+  - path: /var/lib/bootkube/.keep
+  - path: /etc/NetworkManager/conf.d/typhoon.conf
+    content: |
+      [main]
+      plugins=keyfile
+      [keyfile]
+      unmanaged-devices=interface-name:cali*;interface-name:tunl*
+  - path: /etc/selinux/config
+    owner: root:root
+    permissions: '0644'
+    content: |
+      SELINUX=permissive
+      SELINUXTYPE=targeted
+bootcmd:
+  - [setenforce, Permissive]
+  - [systemctl, disable, firewalld, --now]
+  # https://github.com/kubernetes/kubernetes/issues/60869
+  - [modprobe, ip_vs]
+runcmd:
+  - [systemctl, daemon-reload]
+  - [systemctl, restart, NetworkManager]
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.9"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.13.0"
+  - [systemctl, start, --no-block, etcd.service]
+  - [systemctl, enable, cloud-metadata.service]
+  - [systemctl, start, --no-block, kubelet.service]
+users:
+  - default
+  - name: fedora
+    gecos: Fedora Admin
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    groups: wheel,adm,systemd-journal,docker
+    ssh-authorized-keys:
+      - "${ssh_authorized_key}"

aws/fedora-atomic/kubernetes/controllers.tf

@@ -0,0 +1,75 @@
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "aws_route53_record" "etcds" {
+  count = "${var.controller_count}"
+
+  # DNS Zone where record should be created
+  zone_id = "${var.dns_zone_id}"
+
+  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)}"
+  type = "A"
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  records = ["${element(aws_instance.controllers.*.private_ip, count.index)}"]
+}
+
+# Controller instances
+resource "aws_instance" "controllers" {
+  count = "${var.controller_count}"
+
+  tags = {
+    Name = "${var.cluster_name}-controller-${count.index}"
+  }
+
+  instance_type = "${var.controller_type}"
+
+  ami       = "${data.aws_ami.fedora.image_id}"
+  user_data = "${element(data.template_file.controller-cloudinit.*.rendered, count.index)}"
+
+  # storage
+  root_block_device {
+    volume_type = "${var.disk_type}"
+    volume_size = "${var.disk_size}"
+  }
+
+  # network
+  associate_public_ip_address = true
+  subnet_id                   = "${element(aws_subnet.public.*.id, count.index)}"
+  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
+
+  lifecycle {
+    ignore_changes = ["ami"]
+  }
+}
+
+# Controller Cloud-Init
+data "template_file" "controller-cloudinit" {
+  count = "${var.controller_count}"
+
+  template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
+
+    kubeconfig            = "${indent(6, module.bootkube.kubeconfig)}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  }
+}
+
+data "template_file" "etcds" {
+  count    = "${var.controller_count}"
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+  vars {
+    index        = "${count.index}"
+    cluster_name = "${var.cluster_name}"
+    dns_zone     = "${var.dns_zone}"
+  }
+}

aws/fedora-atomic/kubernetes/network.tf

@@ -0,0 +1,57 @@
+data "aws_availability_zones" "all" {}
+
+# Network VPC, gateway, and routes
+
+resource "aws_vpc" "network" {
+  cidr_block                       = "${var.host_cidr}"
+  assign_generated_ipv6_cidr_block = true
+  enable_dns_support               = true
+  enable_dns_hostnames             = true
+
+  tags = "${map("Name", "${var.cluster_name}")}"
+}
+
+resource "aws_internet_gateway" "gateway" {
+  vpc_id = "${aws_vpc.network.id}"
+
+  tags = "${map("Name", "${var.cluster_name}")}"
+}
+
+resource "aws_route_table" "default" {
+  vpc_id = "${aws_vpc.network.id}"
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = "${aws_internet_gateway.gateway.id}"
+  }
+
+  route {
+    ipv6_cidr_block = "::/0"
+    gateway_id      = "${aws_internet_gateway.gateway.id}"
+  }
+
+  tags = "${map("Name", "${var.cluster_name}")}"
+}
+
+# Subnets (one per availability zone)
+
+resource "aws_subnet" "public" {
+  count = "${length(data.aws_availability_zones.all.names)}"
+
+  vpc_id            = "${aws_vpc.network.id}"
+  availability_zone = "${data.aws_availability_zones.all.names[count.index]}"
+
+  cidr_block                      = "${cidrsubnet(var.host_cidr, 4, count.index)}"
+  ipv6_cidr_block                 = "${cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)}"
+  map_public_ip_on_launch         = true
+  assign_ipv6_address_on_creation = true
+
+  tags = "${map("Name", "${var.cluster_name}-public-${count.index}")}"
+}
+
+resource "aws_route_table_association" "public" {
+  count = "${length(data.aws_availability_zones.all.names)}"
+
+  route_table_id = "${aws_route_table.default.id}"
+  subnet_id      = "${element(aws_subnet.public.*.id, count.index)}"
+}

aws/fedora-atomic/kubernetes/nlb.tf

@@ -0,0 +1,93 @@
+# Network Load Balancer DNS Record
+resource "aws_route53_record" "apiserver" {
+  zone_id = "${var.dns_zone_id}"
+
+  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
+  type = "A"
+
+  # AWS recommends their special "alias" records for NLBs
+  alias {
+    name                   = "${aws_lb.nlb.dns_name}"
+    zone_id                = "${aws_lb.nlb.zone_id}"
+    evaluate_target_health = true
+  }
+}
+
+# Network Load Balancer for apiservers and ingress
+resource "aws_lb" "nlb" {
+  name               = "${var.cluster_name}-nlb"
+  load_balancer_type = "network"
+  internal           = false
+
+  subnets = ["${aws_subnet.public.*.id}"]
+
+  enable_cross_zone_load_balancing = true
+}
+
+# Forward TCP apiserver traffic to controllers
+resource "aws_lb_listener" "apiserver-https" {
+  load_balancer_arn = "${aws_lb.nlb.arn}"
+  protocol          = "TCP"
+  port              = "6443"
+
+  default_action {
+    type             = "forward"
+    target_group_arn = "${aws_lb_target_group.controllers.arn}"
+  }
+}
+
+# Forward HTTP ingress traffic to workers
+resource "aws_lb_listener" "ingress-http" {
+  load_balancer_arn = "${aws_lb.nlb.arn}"
+  protocol          = "TCP"
+  port              = 80
+
+  default_action {
+    type             = "forward"
+    target_group_arn = "${module.workers.target_group_http}"
+  }
+}
+
+# Forward HTTPS ingress traffic to workers
+resource "aws_lb_listener" "ingress-https" {
+  load_balancer_arn = "${aws_lb.nlb.arn}"
+  protocol          = "TCP"
+  port              = 443
+
+  default_action {
+    type             = "forward"
+    target_group_arn = "${module.workers.target_group_https}"
+  }
+}
+
+# Target group of controllers
+resource "aws_lb_target_group" "controllers" {
+  name        = "${var.cluster_name}-controllers"
+  vpc_id      = "${aws_vpc.network.id}"
+  target_type = "instance"
+
+  protocol = "TCP"
+  port     = 6443
+
+  # TCP health check for apiserver
+  health_check {
+    protocol = "TCP"
+    port     = 6443
+
+    # NLBs required to use same healthy and unhealthy thresholds
+    healthy_threshold   = 3
+    unhealthy_threshold = 3
+
+    # Interval between health checks required to be 10 or 30
+    interval = 10
+  }
+}
+
+# Attach controller instances to apiserver NLB
+resource "aws_lb_target_group_attachment" "controllers" {
+  count = "${var.controller_count}"
+
+  target_group_arn = "${aws_lb_target_group.controllers.arn}"
+  target_id        = "${element(aws_instance.controllers.*.id, count.index)}"
+  port             = 6443
+}

aws/fedora-atomic/kubernetes/outputs.tf

@@ -0,0 +1,39 @@
+# Outputs for Kubernetes Ingress
+
+output "ingress_dns_name" {
+  value       = "${aws_lb.nlb.dns_name}"
+  description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
+}
+
+# Outputs for worker pools
+
+output "vpc_id" {
+  value       = "${aws_vpc.network.id}"
+  description = "ID of the VPC for creating worker instances"
+}
+
+output "subnet_ids" {
+  value       = ["${aws_subnet.public.*.id}"]
+  description = "List of subnet IDs for creating worker instances"
+}
+
+output "worker_security_groups" {
+  value       = ["${aws_security_group.worker.id}"]
+  description = "List of worker security group IDs"
+}
+
+output "kubeconfig" {
+  value = "${module.bootkube.kubeconfig}"
+}
+
+# Outputs for custom load balancing
+
+output "worker_target_group_http" {
+  description = "ARN of a target group of workers for HTTP traffic"
+  value       = "${module.workers.target_group_http}"
+}
+
+output "worker_target_group_https" {
+  description = "ARN of a target group of workers for HTTPS traffic"
+  value       = "${module.workers.target_group_https}"
+}

aws/fedora-atomic/kubernetes/require.tf

@@ -0,0 +1,25 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = ">= 0.11.0"
+}
+
+provider "aws" {
+  version = "~> 1.13"
+}
+
+provider "local" {
+  version = "~> 1.0"
+}
+
+provider "null" {
+  version = "~> 1.0"
+}
+
+provider "template" {
+  version = "~> 1.0"
+}
+
+provider "tls" {
+  version = "~> 1.0"
+}

aws/fedora-atomic/kubernetes/security.tf

@@ -0,0 +1,393 @@
+# Security Groups (instance firewalls)
+
+# Controller security group
+
+resource "aws_security_group" "controller" {
+  name        = "${var.cluster_name}-controller"
+  description = "${var.cluster_name} controller security group"
+
+  vpc_id = "${aws_vpc.network.id}"
+
+  tags = "${map("Name", "${var.cluster_name}-controller")}"
+}
+
+resource "aws_security_group_rule" "controller-ssh" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 22
+  to_port     = 22
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
+resource "aws_security_group_rule" "controller-etcd" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 2379
+  to_port   = 2380
+  self      = true
+}
+
+# Allow Prometheus to scrape etcd metrics
+resource "aws_security_group_rule" "controller-etcd-metrics" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 2381
+  to_port                  = 2381
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-apiserver" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 6443
+  to_port     = 6443
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
+resource "aws_security_group_rule" "controller-flannel" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "udp"
+  from_port                = 8472
+  to_port                  = 8472
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-flannel-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = "udp"
+  from_port = 8472
+  to_port   = 8472
+  self      = true
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "aws_security_group_rule" "controller-node-exporter" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 9100
+  to_port                  = 9100
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+# Allow apiserver to access kubelets for exec, log, port-forward
+resource "aws_security_group_rule" "controller-kubelet" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10250
+  to_port                  = 10250
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-kubelet-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10250
+  to_port   = 10250
+  self      = true
+}
+
+# Allow heapster / metrics-server to scrape kubelet read-only
+resource "aws_security_group_rule" "controller-kubelet-read" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10255
+  to_port                  = 10255
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-kubelet-read-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10255
+  to_port   = 10255
+  self      = true
+}
+
+resource "aws_security_group_rule" "controller-bgp" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 179
+  to_port                  = 179
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-bgp-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 179
+  to_port   = 179
+  self      = true
+}
+
+resource "aws_security_group_rule" "controller-ipip" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = 4
+  from_port                = 0
+  to_port                  = 0
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-ipip-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = 4
+  from_port = 0
+  to_port   = 0
+  self      = true
+}
+
+resource "aws_security_group_rule" "controller-ipip-legacy" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = 94
+  from_port                = 0
+  to_port                  = 0
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
+resource "aws_security_group_rule" "controller-ipip-legacy-self" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type      = "ingress"
+  protocol  = 94
+  from_port = 0
+  to_port   = 0
+  self      = true
+}
+
+resource "aws_security_group_rule" "controller-egress" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type             = "egress"
+  protocol         = "-1"
+  from_port        = 0
+  to_port          = 0
+  cidr_blocks      = ["0.0.0.0/0"]
+  ipv6_cidr_blocks = ["::/0"]
+}
+
+# Worker security group
+
+resource "aws_security_group" "worker" {
+  name        = "${var.cluster_name}-worker"
+  description = "${var.cluster_name} worker security group"
+
+  vpc_id = "${aws_vpc.network.id}"
+
+  tags = "${map("Name", "${var.cluster_name}-worker")}"
+}
+
+resource "aws_security_group_rule" "worker-ssh" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 22
+  to_port     = 22
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
+resource "aws_security_group_rule" "worker-http" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 80
+  to_port     = 80
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
+resource "aws_security_group_rule" "worker-https" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 443
+  to_port     = 443
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
+resource "aws_security_group_rule" "worker-flannel" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type                     = "ingress"
+  protocol                 = "udp"
+  from_port                = 8472
+  to_port                  = 8472
+  source_security_group_id = "${aws_security_group.controller.id}"
+}
+
+resource "aws_security_group_rule" "worker-flannel-self" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = "udp"
+  from_port = 8472
+  to_port   = 8472
+  self      = true
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "aws_security_group_rule" "worker-node-exporter" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 9100
+  to_port   = 9100
+  self      = true
+}
+
+resource "aws_security_group_rule" "ingress-health" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 10254
+  to_port     = 10254
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
+# Allow apiserver to access kubelets for exec, log, port-forward
+resource "aws_security_group_rule" "worker-kubelet" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10250
+  to_port                  = 10250
+  source_security_group_id = "${aws_security_group.controller.id}"
+}
+
+# Allow Prometheus to scrape kubelet metrics
+resource "aws_security_group_rule" "worker-kubelet-self" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10250
+  to_port   = 10250
+  self      = true
+}
+
+# Allow heapster / metrics-server to scrape kubelet read-only
+resource "aws_security_group_rule" "worker-kubelet-read" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10255
+  to_port                  = 10255
+  source_security_group_id = "${aws_security_group.controller.id}"
+}
+
+resource "aws_security_group_rule" "worker-kubelet-read-self" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10255
+  to_port   = 10255
+  self      = true
+}
+
+resource "aws_security_group_rule" "worker-bgp" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 179
+  to_port                  = 179
+  source_security_group_id = "${aws_security_group.controller.id}"
+}
+
+resource "aws_security_group_rule" "worker-bgp-self" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 179
+  to_port   = 179
+  self      = true
+}
+
+resource "aws_security_group_rule" "worker-ipip" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type                     = "ingress"
+  protocol                 = 4
+  from_port                = 0
+  to_port                  = 0
+  source_security_group_id = "${aws_security_group.controller.id}"
+}
+
+resource "aws_security_group_rule" "worker-ipip-self" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = 4
+  from_port = 0
+  to_port   = 0
+  self      = true
+}
+
+resource "aws_security_group_rule" "worker-ipip-legacy" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type                     = "ingress"
+  protocol                 = 94
+  from_port                = 0
+  to_port                  = 0
+  source_security_group_id = "${aws_security_group.controller.id}"
+}
+
+resource "aws_security_group_rule" "worker-ipip-legacy-self" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type      = "ingress"
+  protocol  = 94
+  from_port = 0
+  to_port   = 0
+  self      = true
+}
+
+resource "aws_security_group_rule" "worker-egress" {
+  security_group_id = "${aws_security_group.worker.id}"
+
+  type             = "egress"
+  protocol         = "-1"
+  from_port        = 0
+  to_port          = 0
+  cidr_blocks      = ["0.0.0.0/0"]
+  ipv6_cidr_blocks = ["::/0"]
+}

aws/fedora-atomic/kubernetes/ssh.tf

@@ -0,0 +1,89 @@
+# Secure copy etcd TLS assets to controllers.
+resource "null_resource" "copy-controller-secrets" {
+  count = "${var.controller_count}"
+
+  connection {
+    type    = "ssh"
+    host    = "${element(aws_instance.controllers.*.public_ip, count.index)}"
+    user    = "fedora"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_ca_cert}"
+    destination = "$HOME/etcd-client-ca.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_cert}"
+    destination = "$HOME/etcd-client.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_key}"
+    destination = "$HOME/etcd-client.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_cert}"
+    destination = "$HOME/etcd-server.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_key}"
+    destination = "$HOME/etcd-server.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_cert}"
+    destination = "$HOME/etcd-peer.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_key}"
+    destination = "$HOME/etcd-peer.key"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mkdir -p /etc/ssl/etcd/etcd",
+      "sudo mv etcd-client* /etc/ssl/etcd/",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
+      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
+      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
+      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
+      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
+    ]
+  }
+}
+
+# Secure copy bootkube assets to ONE controller and start bootkube to perform
+# one-time self-hosted cluster bootstrapping.
+resource "null_resource" "bootkube-start" {
+  depends_on = [
+    "null_resource.copy-controller-secrets",
+    "module.workers",
+    "aws_route53_record.apiserver",
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = "${aws_instance.controllers.0.public_ip}"
+    user    = "fedora"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    source      = "${var.asset_dir}"
+    destination = "$HOME/assets"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do sleep 4; done",
+      "sudo mv $HOME/assets /var/lib/bootkube",
+      "sudo systemctl start bootkube",
+    ]
+  }
+}

aws/fedora-atomic/kubernetes/variables.tf

@@ -0,0 +1,112 @@
+variable "cluster_name" {
+  type        = "string"
+  description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# AWS
+
+variable "dns_zone" {
+  type        = "string"
+  description = "AWS DNS Zone (e.g. aws.example.com)"
+}
+
+variable "dns_zone_id" {
+  type        = "string"
+  description = "AWS DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
+}
+
+# instances
+
+variable "controller_count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of controllers (i.e. masters)"
+}
+
+variable "worker_count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of workers"
+}
+
+variable "controller_type" {
+  type        = "string"
+  default     = "t2.small"
+  description = "EC2 instance type for controllers"
+}
+
+variable "worker_type" {
+  type        = "string"
+  default     = "t2.small"
+  description = "EC2 instance type for workers"
+}
+
+variable "disk_size" {
+  type        = "string"
+  default     = "40"
+  description = "Size of the EBS volume in GB"
+}
+
+variable "disk_type" {
+  type        = "string"
+  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
+}
+
+variable "worker_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+  type        = "string"
+  description = "SSH public key for user 'fedora'"
+}
+
+variable "asset_dir" {
+  description = "Path to a directory where generated assets should be placed (contains secrets)"
+  type        = "string"
+}
+
+variable "networking" {
+  description = "Choice of networking provider (calico or flannel)"
+  type        = "string"
+  default     = "calico"
+}
+
+variable "network_mtu" {
+  description = "CNI interface MTU (applies to calico only). Use 8981 if using instances types with Jumbo frames."
+  type        = "string"
+  default     = "1480"
+}
+
+variable "host_cidr" {
+  description = "CIDR IPv4 range to assign to EC2 nodes"
+  type        = "string"
+  default     = "10.0.0.0/16"
+}
+
+variable "pod_cidr" {
+  description = "CIDR IPv4 range to assign Kubernetes pods"
+  type        = "string"
+  default     = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+
+  type    = "string"
+  default = "10.3.0.0/16"
+}
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}

aws/fedora-atomic/kubernetes/workers.tf

@@ -0,0 +1,19 @@
+module "workers" {
+  source = "workers"
+  name   = "${var.cluster_name}"
+
+  # AWS
+  vpc_id          = "${aws_vpc.network.id}"
+  subnet_ids      = ["${aws_subnet.public.*.id}"]
+  security_groups = ["${aws_security_group.worker.id}"]
+  count           = "${var.worker_count}"
+  instance_type   = "${var.worker_type}"
+  disk_size       = "${var.disk_size}"
+  spot_price      = "${var.worker_price}"
+
+  # configuration
+  kubeconfig            = "${module.bootkube.kubeconfig}"
+  ssh_authorized_key    = "${var.ssh_authorized_key}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+}

aws/fedora-atomic/kubernetes/workers/ami.tf

@@ -0,0 +1,19 @@
+data "aws_ami" "fedora" {
+  most_recent = true
+  owners      = ["125523088429"]
+
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Fedora-AtomicHost-28-20180625.1.x86_64-*-gp2-*"]
+  }
+}

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -0,0 +1,81 @@
+#cloud-config
+write_files:
+  - path: /etc/systemd/system/cloud-metadata.service
+    content: |
+      [Unit]
+      Description=Cloud metadata agent
+      [Service]
+      Type=oneshot
+      Environment=OUTPUT=/run/metadata/cloud
+      ExecStart=/usr/bin/mkdir -p /run/metadata
+      ExecStart=/usr/bin/bash -c 'echo "HOSTNAME_OVERRIDE=$(curl\
+        --url http://169.254.169.254/latest/meta-data/local-ipv4\
+        --retry 10)" > $${OUTPUT}'
+      [Install]
+      WantedBy=multi-user.target
+  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
+    content: |
+      [Unit]
+      Requires=cloud-metadata.service
+      After=cloud-metadata.service
+      Wants=rpc-statd.service
+      [Service]
+      ExecStartPre=/bin/mkdir -p /opt/cni/bin
+      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+      ExecStartPre=/bin/mkdir -p /var/lib/cni
+      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+      Restart=always
+      RestartSec=10
+  - path: /etc/kubernetes/kubelet.conf
+    content: |
+      ARGS="--anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
+        --client-ca-file=/etc/kubernetes/ca.crt \
+        --cluster_dns=${k8s_dns_service_ip} \
+        --cluster_domain=${cluster_domain_suffix} \
+        --cni-conf-dir=/etc/kubernetes/cni/net.d \
+        --exit-on-lock-contention \
+        --kubeconfig=/etc/kubernetes/kubeconfig \
+        --lock-file=/var/run/lock/kubelet.lock \
+        --network-plugin=cni \
+        --node-labels=node-role.kubernetes.io/node \
+        --pod-manifest-path=/etc/kubernetes/manifests \
+        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
+  - path: /etc/kubernetes/kubeconfig
+    permissions: '0644'
+    content: |
+      ${kubeconfig}
+  - path: /etc/NetworkManager/conf.d/typhoon.conf
+    content: |
+      [main]
+      plugins=keyfile
+      [keyfile]
+      unmanaged-devices=interface-name:cali*;interface-name:tunl*
+  - path: /etc/selinux/config
+    owner: root:root
+    permissions: '0644'
+    content: |
+      SELINUX=permissive
+      SELINUXTYPE=targeted
+bootcmd:
+  - [setenforce, Permissive]
+  - [systemctl, disable, firewalld, --now]
+  # https://github.com/kubernetes/kubernetes/issues/60869
+  - [modprobe, ip_vs]
+runcmd:
+  - [systemctl, daemon-reload]
+  - [systemctl, restart, NetworkManager]
+  - [systemctl, enable, cloud-metadata.service]
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.11.3"
+  - [systemctl, start, --no-block, kubelet.service]
+users:
+  - default
+  - name: fedora
+    gecos: Fedora Admin
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    groups: wheel,adm,systemd-journal,docker
+    ssh-authorized-keys:
+      - "${ssh_authorized_key}"

aws/fedora-atomic/kubernetes/workers/ingress.tf

@@ -0,0 +1,47 @@
+# Target groups of instances for use with load balancers
+
+resource "aws_lb_target_group" "workers-http" {
+  name        = "${var.name}-workers-http"
+  vpc_id      = "${var.vpc_id}"
+  target_type = "instance"
+
+  protocol = "TCP"
+  port     = 80
+
+  # HTTP health check for ingress
+  health_check {
+    protocol = "HTTP"
+    port     = 10254
+    path     = "/healthz"
+
+    # NLBs required to use same healthy and unhealthy thresholds
+    healthy_threshold   = 3
+    unhealthy_threshold = 3
+
+    # Interval between health checks required to be 10 or 30
+    interval = 10
+  }
+}
+
+resource "aws_lb_target_group" "workers-https" {
+  name        = "${var.name}-workers-https"
+  vpc_id      = "${var.vpc_id}"
+  target_type = "instance"
+
+  protocol = "TCP"
+  port     = 443
+
+  # HTTP health check for ingress
+  health_check {
+    protocol = "HTTP"
+    port     = 10254
+    path     = "/healthz"
+
+    # NLBs required to use same healthy and unhealthy thresholds
+    healthy_threshold   = 3
+    unhealthy_threshold = 3
+
+    # Interval between health checks required to be 10 or 30
+    interval = 10
+  }
+}

aws/fedora-atomic/kubernetes/workers/outputs.tf

@@ -0,0 +1,9 @@
+output "target_group_http" {
+  description = "ARN of a target group of workers for HTTP traffic"
+  value       = "${aws_lb_target_group.workers-http.arn}"
+}
+
+output "target_group_https" {
+  description = "ARN of a target group of workers for HTTPS traffic"
+  value       = "${aws_lb_target_group.workers-https.arn}"
+}

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -0,0 +1,81 @@
+variable "name" {
+  type        = "string"
+  description = "Unique name for the worker pool"
+}
+
+# AWS
+
+variable "vpc_id" {
+  type        = "string"
+  description = "Must be set to `vpc_id` output by cluster"
+}
+
+variable "subnet_ids" {
+  type        = "list"
+  description = "Must be set to `subnet_ids` output by cluster"
+}
+
+variable "security_groups" {
+  type        = "list"
+  description = "Must be set to `worker_security_groups` output by cluster"
+}
+
+# instances
+
+variable "count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of instances"
+}
+
+variable "instance_type" {
+  type        = "string"
+  default     = "t2.small"
+  description = "EC2 instance type"
+}
+
+variable "disk_size" {
+  type        = "string"
+  default     = "40"
+  description = "Size of the EBS volume in GB"
+}
+
+variable "disk_type" {
+  type        = "string"
+  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
+}
+
+variable "spot_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
+# configuration
+
+variable "kubeconfig" {
+  type        = "string"
+  description = "Must be set to `kubeconfig` output by cluster"
+}
+
+variable "ssh_authorized_key" {
+  type        = "string"
+  description = "SSH public key for user 'fedora'"
+}
+
+variable "service_cidr" {
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+
+  type    = "string"
+  default = "10.3.0.0/16"
+}
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -0,0 +1,77 @@
+# Workers AutoScaling Group
+resource "aws_autoscaling_group" "workers" {
+  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
+
+  # count
+  desired_capacity          = "${var.count}"
+  min_size                  = "${var.count}"
+  max_size                  = "${var.count + 2}"
+  default_cooldown          = 30
+  health_check_grace_period = 30
+
+  # network
+  vpc_zone_identifier = ["${var.subnet_ids}"]
+
+  # template
+  launch_configuration = "${aws_launch_configuration.worker.name}"
+
+  # target groups to which instances should be added
+  target_group_arns = [
+    "${aws_lb_target_group.workers-http.id}",
+    "${aws_lb_target_group.workers-https.id}",
+  ]
+
+  lifecycle {
+    # override the default destroy and replace update behavior
+    create_before_destroy = true
+  }
+
+  # Waiting for instance creation delays adding the ASG to state. If instances
+  # can't be created (e.g. spot price too low), the ASG will be orphaned.
+  # Orphaned ASGs escape cleanup, can't be updated, and keep bidding if spot is
+  # used. Disable wait to avoid issues and align with other clouds.
+  wait_for_capacity_timeout = "0"
+
+  tags = [{
+    key                 = "Name"
+    value               = "${var.name}-worker"
+    propagate_at_launch = true
+  }]
+}
+
+# Worker template
+resource "aws_launch_configuration" "worker" {
+  image_id          = "${data.aws_ami.fedora.image_id}"
+  instance_type     = "${var.instance_type}"
+  spot_price        = "${var.spot_price}"
+  enable_monitoring = false
+
+  user_data = "${data.template_file.worker-cloudinit.rendered}"
+
+  # storage
+  root_block_device {
+    volume_type = "${var.disk_type}"
+    volume_size = "${var.disk_size}"
+  }
+
+  # network
+  security_groups = ["${var.security_groups}"]
+
+  lifecycle {
+    // Override the default destroy and replace update behavior
+    create_before_destroy = true
+    ignore_changes        = ["image_id"]
+  }
+}
+
+# Worker Cloud-Init
+data "template_file" "worker-cloudinit" {
+  template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
+
+  vars = {
+    kubeconfig            = "${indent(6, var.kubeconfig)}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  }
+}

azure/container-linux/kubernetes/LICENSE

@@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Typhoon Authors
+Copyright (c) 2017 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

azure/container-linux/kubernetes/README.md

@@ -0,0 +1,22 @@
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
+
+* Kubernetes v1.11.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the Azure [tutorial](https://typhoon.psdn.io/cl/azure/).
+

azure/container-linux/kubernetes/bootkube.tf

@@ -0,0 +1,13 @@
+# Self-hosted Kubernetes assets (kubeconfig, manifests)
+module "bootkube" {
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=5378e166ef7ec44e69fbc2d879dbf048a45a0d09"
+
+  cluster_name          = "${var.cluster_name}"
+  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
+  etcd_servers          = ["${formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)}"]
+  asset_dir             = "${var.asset_dir}"
+  networking            = "flannel"
+  pod_cidr              = "${var.pod_cidr}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+}

azure/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -0,0 +1,163 @@
+---
+systemd:
+  units:
+    - name: etcd-member.service
+      enable: true
+      dropins:
+        - name: 40-etcd-cluster.conf
+          contents: |
+            [Service]
+            Environment="ETCD_IMAGE_TAG=v3.3.9"
+            Environment="ETCD_NAME=${etcd_name}"
+            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
+            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
+            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
+            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
+            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
+            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
+            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
+            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
+            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
+            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
+            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
+            Environment="ETCD_CLIENT_CERT_AUTH=true"
+            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
+            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
+            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
+            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
+    - name: docker.service
+      enable: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube
+        Wants=rpc-statd.service
+        [Service]
+        EnvironmentFile=/etc/kubernetes/kubelet.env
+        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --volume=resolv,kind=host,source=/etc/resolv.conf \
+          --mount volume=resolv,target=/etc/resolv.conf \
+          --volume var-lib-cni,kind=host,source=/var/lib/cni \
+          --mount volume=var-lib-cni,target=/var/lib/cni \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --insecure-options=image"
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
+        ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
+        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${k8s_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node-role.kubernetes.io/master \
+          --node-labels=node-role.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootkube.service
+      contents: |
+        [Unit]
+        Description=Bootstrap a Kubernetes cluster
+        ConditionPathExists=!/opt/bootkube/init_bootkube.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootkube
+        ExecStart=/opt/bootkube/bootkube-start
+        ExecStartPost=/bin/touch /opt/bootkube/init_bootkube.done
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.env
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
+          KUBELET_IMAGE_TAG=v1.11.3
+    - path: /etc/sysctl.d/max-user-watches.conf
+      filesystem: root
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /opt/bootkube/bootkube-start
+      filesystem: root
+      mode: 0544
+      user:
+        id: 500
+      group:
+        id: 500
+      contents:
+        inline: |
+          #!/bin/bash
+          # Wrapper for bootkube start
+          set -e
+          # Move experimental manifests
+          [ -n "$(ls /opt/bootkube/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootkube/assets/manifests-*/* /opt/bootkube/assets/manifests && rm -rf /opt/bootkube/assets/manifests-*
+          BOOTKUBE_ACI="$${BOOTKUBE_ACI:-quay.io/coreos/bootkube}"
+          BOOTKUBE_VERSION="$${BOOTKUBE_VERSION:-v0.13.0}"
+          BOOTKUBE_ASSETS="$${BOOTKUBE_ASSETS:-/opt/bootkube/assets}"
+          exec /usr/bin/rkt run \
+            --trust-keys-from-https \
+            --volume assets,kind=host,source=$${BOOTKUBE_ASSETS} \
+            --mount volume=assets,target=/assets \
+            --volume bootstrap,kind=host,source=/etc/kubernetes \
+            --mount volume=bootstrap,target=/etc/kubernetes \
+            $${RKT_OPTS} \
+            $${BOOTKUBE_ACI}:$${BOOTKUBE_VERSION} \
+            --net=host \
+            --dns=host \
+            --exec=/bootkube -- start --asset-dir=/assets "$@"
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"

azure/container-linux/kubernetes/controllers.tf

@@ -0,0 +1,163 @@
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "azurerm_dns_a_record" "etcds" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${var.dns_zone_group}"
+
+  # DNS Zone name where record should be created
+  zone_name = "${var.dns_zone}"
+
+  # DNS record
+  name = "${format("%s-etcd%d", var.cluster_name, count.index)}"
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  records = ["${element(azurerm_network_interface.controllers.*.private_ip_address, count.index)}"]
+}
+
+locals {
+  # Channel for a Container Linux derivative
+  # coreos-stable -> Container Linux Stable
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
+# Controller availability set to spread controllers
+resource "azurerm_availability_set" "controllers" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-controllers"
+  location                     = "${var.region}"
+  platform_fault_domain_count  = 2
+  platform_update_domain_count = 4
+  managed                      = true
+}
+
+# Controller instances
+resource "azurerm_virtual_machine" "controllers" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                = "${var.cluster_name}-controller-${count.index}"
+  location            = "${var.region}"
+  availability_set_id = "${azurerm_availability_set.controllers.id}"
+  vm_size             = "${var.controller_type}"
+
+  # boot
+  storage_image_reference {
+    publisher = "CoreOS"
+    offer     = "CoreOS"
+    sku       = "${local.channel}"
+    version   = "latest"
+  }
+
+  # storage
+  storage_os_disk {
+    name              = "${var.cluster_name}-controller-${count.index}"
+    create_option     = "FromImage"
+    caching           = "ReadWrite"
+    disk_size_gb      = "${var.disk_size}"
+    os_type           = "Linux"
+    managed_disk_type = "Premium_LRS"
+  }
+
+  # network
+  network_interface_ids = ["${element(azurerm_network_interface.controllers.*.id, count.index)}"]
+
+  os_profile {
+    computer_name  = "${var.cluster_name}-controller-${count.index}"
+    admin_username = "core"
+    custom_data    = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  }
+
+  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
+  os_profile_linux_config {
+    disable_password_authentication = true
+
+    ssh_keys {
+      path     = "/home/core/.ssh/authorized_keys"
+      key_data = "${var.ssh_authorized_key}"
+    }
+  }
+
+  # lifecycle
+  delete_os_disk_on_termination    = true
+  delete_data_disks_on_termination = true
+
+  lifecycle {
+    ignore_changes = [
+      "storage_os_disk",
+    ]
+  }
+}
+
+# Controller NICs with public and private IPv4
+resource "azurerm_network_interface" "controllers" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                      = "${var.cluster_name}-controller-${count.index}"
+  location                  = "${azurerm_resource_group.cluster.location}"
+  network_security_group_id = "${azurerm_network_security_group.controller.id}"
+
+  ip_configuration {
+    name                          = "ip0"
+    subnet_id                     = "${azurerm_subnet.controller.id}"
+    private_ip_address_allocation = "dynamic"
+
+    # public IPv4
+    public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
+
+    # backend address pool to which the NIC should be added
+    load_balancer_backend_address_pools_ids = ["${azurerm_lb_backend_address_pool.controller.id}"]
+  }
+}
+
+# Controller public IPv4 addresses
+resource "azurerm_public_ip" "controllers" {
+  count               = "${var.controller_count}"
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-controller-${count.index}"
+  location                     = "${azurerm_resource_group.cluster.location}"
+  sku                          = "Standard"
+  public_ip_address_allocation = "static"
+}
+
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = "${var.controller_count}"
+  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  pretty_print = false
+  snippets     = ["${var.controller_clc_snippets}"]
+}
+
+# Controller Container Linux configs
+data "template_file" "controller-configs" {
+  count = "${var.controller_count}"
+
+  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
+
+    kubeconfig            = "${indent(10, module.bootkube.kubeconfig)}"
+    ssh_authorized_key    = "${var.ssh_authorized_key}"
+    k8s_dns_service_ip    = "${cidrhost(var.service_cidr, 10)}"
+    cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  }
+}
+
+data "template_file" "etcds" {
+  count    = "${var.controller_count}"
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+  vars {
+    index        = "${count.index}"
+    cluster_name = "${var.cluster_name}"
+    dns_zone     = "${var.dns_zone}"
+  }
+}

azure/container-linux/kubernetes/lb.tf

@@ -0,0 +1,144 @@
+# DNS record for the apiserver load balancer
+resource "azurerm_dns_a_record" "apiserver" {
+  resource_group_name = "${var.dns_zone_group}"
+
+  # DNS Zone name where record should be created
+  zone_name = "${var.dns_zone}"
+
+  # DNS record
+  name = "${var.cluster_name}"
+  ttl  = 300
+
+  # IPv4 address of apiserver load balancer
+  records = ["${azurerm_public_ip.apiserver-ipv4.ip_address}"]
+}
+
+# Static IPv4 address for the apiserver frontend
+resource "azurerm_public_ip" "apiserver-ipv4" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-apiserver-ipv4"
+  location                     = "${var.region}"
+  sku                          = "Standard"
+  public_ip_address_allocation = "static"
+}
+
+# Static IPv4 address for the ingress frontend
+resource "azurerm_public_ip" "ingress-ipv4" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                         = "${var.cluster_name}-ingress-ipv4"
+  location                     = "${var.region}"
+  sku                          = "Standard"
+  public_ip_address_allocation = "static"
+}
+
+# Network Load Balancer for apiservers and ingress
+resource "azurerm_lb" "cluster" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name     = "${var.cluster_name}"
+  location = "${var.region}"
+  sku      = "Standard"
+
+  frontend_ip_configuration {
+    name                 = "apiserver"
+    public_ip_address_id = "${azurerm_public_ip.apiserver-ipv4.id}"
+  }
+
+  frontend_ip_configuration {
+    name                 = "ingress"
+    public_ip_address_id = "${azurerm_public_ip.ingress-ipv4.id}"
+  }
+}
+
+resource "azurerm_lb_rule" "apiserver" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                           = "apiserver"
+  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  frontend_ip_configuration_name = "apiserver"
+
+  protocol                = "Tcp"
+  frontend_port           = 6443
+  backend_port            = 6443
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+  probe_id                = "${azurerm_lb_probe.apiserver.id}"
+}
+
+resource "azurerm_lb_rule" "ingress-http" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                           = "ingress-http"
+  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  frontend_ip_configuration_name = "ingress"
+
+  protocol                = "Tcp"
+  frontend_port           = 80
+  backend_port            = 80
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  probe_id                = "${azurerm_lb_probe.ingress.id}"
+}
+
+resource "azurerm_lb_rule" "ingress-https" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                           = "ingress-https"
+  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  frontend_ip_configuration_name = "ingress"
+
+  protocol                = "Tcp"
+  frontend_port           = 443
+  backend_port            = 443
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  probe_id                = "${azurerm_lb_probe.ingress.id}"
+}
+
+# Address pool of controllers
+resource "azurerm_lb_backend_address_pool" "controller" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "controller"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+}
+
+# Address pool of workers
+resource "azurerm_lb_backend_address_pool" "worker" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "worker"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+}
+
+# Health checks / probes
+
+# TCP health check for apiserver
+resource "azurerm_lb_probe" "apiserver" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "apiserver"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  protocol        = "Tcp"
+  port            = 6443
+
+  # unhealthy threshold
+  number_of_probes = 3
+
+  interval_in_seconds = 5
+}
+
+# HTTP health check for ingress
+resource "azurerm_lb_probe" "ingress" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name            = "ingress"
+  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  protocol        = "Http"
+  port            = 10254
+  request_path    = "/healthz"
+
+  # unhealthy threshold
+  number_of_probes = 3
+
+  interval_in_seconds = 5
+}

azure/container-linux/kubernetes/network.tf

@@ -0,0 +1,33 @@
+# Organize cluster into a resource group
+resource "azurerm_resource_group" "cluster" {
+  name     = "${var.cluster_name}"
+  location = "${var.region}"
+}
+
+resource "azurerm_virtual_network" "network" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name          = "${var.cluster_name}"
+  location      = "${azurerm_resource_group.cluster.location}"
+  address_space = ["${var.host_cidr}"]
+}
+
+# Subnets - separate subnets for controller and workers because Azure
+# network security groups are based on IPv4 CIDR rather than instance
+# tags like GCP or security group membership like AWS
+
+resource "azurerm_subnet" "controller" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                 = "controller"
+  virtual_network_name = "${azurerm_virtual_network.network.name}"
+  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 0)}"
+}
+
+resource "azurerm_subnet" "worker" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                 = "worker"
+  virtual_network_name = "${azurerm_virtual_network.network.name}"
+  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 1)}"
+}

azure/container-linux/kubernetes/outputs.tf

@@ -0,0 +1,32 @@
+# Outputs for Kubernetes Ingress
+
+output "ingress_static_ipv4" {
+  value       = "${azurerm_public_ip.ingress-ipv4.ip_address}"
+  description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
+}
+
+# Outputs for worker pools
+
+output "region" {
+  value = "${azurerm_resource_group.cluster.location}"
+}
+
+output "resource_group_name" {
+  value = "${azurerm_resource_group.cluster.name}"
+}
+
+output "subnet_id" {
+  value = "${azurerm_subnet.worker.id}"
+}
+
+output "security_group_id" {
+  value = "${azurerm_network_security_group.worker.id}"
+}
+
+output "backend_address_pool_id" {
+  value = "${azurerm_lb_backend_address_pool.worker.id}"
+}
+
+output "kubeconfig" {
+  value = "${module.bootkube.kubeconfig}"
+}

azure/container-linux/kubernetes/security.tf

@@ -0,0 +1,319 @@
+# Controller security group
+
+resource "azurerm_network_security_group" "controller" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name     = "${var.cluster_name}-controller"
+  location = "${azurerm_resource_group.cluster.location}"
+}
+
+resource "azurerm_network_security_rule" "controller-ssh" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-ssh"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "22"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "controller-etcd" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-etcd"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2005"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "2379-2380"
+  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow Prometheus to scrape etcd metrics
+resource "azurerm_network_security_rule" "controller-etcd-metrics" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-etcd-metrics"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2010"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "2381"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "controller-apiserver" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-apiserver"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2015"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "6443"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "controller-flannel" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-flannel"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2020"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Udp"
+  source_port_range           = "*"
+  destination_port_range      = "8472"
+  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "controller-node-exporter" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-node-exporter"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2025"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "9100"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "controller-kubelet" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-kubelet"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2030"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10250"
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Allow heapster / metrics-server to scrape kubelet read-only
+resource "azurerm_network_security_rule" "controller-kubelet-read" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-kubelet-read"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "2035"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10255"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-loadbalancer"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "3000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "AzureLoadBalancer"
+  destination_address_prefix  = "*"
+}
+
+resource "azurerm_network_security_rule" "controller-deny-all" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "deny-all"
+  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  priority                    = "3005"
+  access                      = "Deny"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "*"
+}
+
+# Worker security group
+
+resource "azurerm_network_security_group" "worker" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name     = "${var.cluster_name}-worker"
+  location = "${azurerm_resource_group.cluster.location}"
+}
+
+resource "azurerm_network_security_rule" "worker-ssh" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-ssh"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "22"
+  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "worker-http" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-http"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2005"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "80"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "worker-https" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-https"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2010"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "443"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+resource "azurerm_network_security_rule" "worker-flannel" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-flannel"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2015"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Udp"
+  source_port_range           = "*"
+  destination_port_range      = "8472"
+  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "azurerm_network_security_rule" "worker-node-exporter" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-node-exporter"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2020"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "9100"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Allow apiserver to access kubelet's for exec, log, port-forward
+resource "azurerm_network_security_rule" "worker-kubelet" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-kubelet"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2025"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10250"
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
+  destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Allow heapster / metrics-server to scrape kubelet read-only
+resource "azurerm_network_security_rule" "worker-kubelet-read" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-kubelet-read"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "2030"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10255"
+  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+}
+
+# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
+# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
+
+resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "allow-loadbalancer"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "3000"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "AzureLoadBalancer"
+  destination_address_prefix  = "*"
+}
+
+resource "azurerm_network_security_rule" "worker-deny-all" {
+  resource_group_name = "${azurerm_resource_group.cluster.name}"
+
+  name                        = "deny-all"
+  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  priority                    = "3005"
+  access                      = "Deny"
+  direction                   = "Inbound"
+  protocol                    = "*"
+  source_port_range           = "*"
+  destination_port_range      = "*"
+  source_address_prefix       = "*"
+  destination_address_prefix  = "*"
+}

azure/container-linux/kubernetes/ssh.tf

@@ -0,0 +1,95 @@
+# Secure copy etcd TLS assets to controllers.
+resource "null_resource" "copy-controller-secrets" {
+  count = "${var.controller_count}"
+
+  depends_on = [
+    "azurerm_virtual_machine.controllers",
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = "${element(azurerm_public_ip.controllers.*.ip_address, count.index)}"
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_ca_cert}"
+    destination = "$HOME/etcd-client-ca.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_cert}"
+    destination = "$HOME/etcd-client.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_client_key}"
+    destination = "$HOME/etcd-client.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_cert}"
+    destination = "$HOME/etcd-server.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_server_key}"
+    destination = "$HOME/etcd-server.key"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_cert}"
+    destination = "$HOME/etcd-peer.crt"
+  }
+
+  provisioner "file" {
+    content     = "${module.bootkube.etcd_peer_key}"
+    destination = "$HOME/etcd-peer.key"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mkdir -p /etc/ssl/etcd/etcd",
+      "sudo mv etcd-client* /etc/ssl/etcd/",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
+      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
+      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
+      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
+      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
+      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
+      "sudo chown -R etcd:etcd /etc/ssl/etcd",
+      "sudo chmod -R 500 /etc/ssl/etcd",
+    ]
+  }
+}
+
+# Secure copy bootkube assets to ONE controller and start bootkube to perform
+# one-time self-hosted cluster bootstrapping.
+resource "null_resource" "bootkube-start" {
+  depends_on = [
+    "module.bootkube",
+    "module.workers",
+    "azurerm_dns_a_record.apiserver",
+    "null_resource.copy-controller-secrets",
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = "${element(azurerm_public_ip.controllers.*.ip_address, 0)}"
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    source      = "${var.asset_dir}"
+    destination = "$HOME/assets"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mv $HOME/assets /opt/bootkube",
+      "sudo systemctl start bootkube",
+    ]
+  }
+}

azure/container-linux/kubernetes/variables.tf

@@ -0,0 +1,117 @@
+variable "cluster_name" {
+  type        = "string"
+  description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# Azure
+
+variable "region" {
+  type        = "string"
+  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
+}
+
+variable "dns_zone" {
+  type        = "string"
+  description = "Azure DNS Zone (e.g. azure.example.com)"
+}
+
+variable "dns_zone_group" {
+  type        = "string"
+  description = "Resource group where the Azure DNS Zone resides (e.g. global)"
+}
+
+# instances
+
+variable "controller_count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of controllers (i.e. masters)"
+}
+
+variable "worker_count" {
+  type        = "string"
+  default     = "1"
+  description = "Number of workers"
+}
+
+variable "controller_type" {
+  type        = "string"
+  default     = "Standard_DS1_v2"
+  description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
+}
+
+variable "worker_type" {
+  type        = "string"
+  default     = "Standard_F1"
+  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
+}
+
+variable "os_image" {
+  type        = "string"
+  default     = "coreos-stable"
+  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
+}
+
+variable "disk_size" {
+  type        = "string"
+  default     = "40"
+  description = "Size of the disk in GB"
+}
+
+variable "worker_priority" {
+  type        = "string"
+  default     = "Regular"
+  description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
+}
+
+variable "controller_clc_snippets" {
+  type        = "list"
+  description = "Controller Container Linux Config snippets"
+  default     = []
+}
+
+variable "worker_clc_snippets" {
+  type        = "list"
+  description = "Worker Container Linux Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+  type        = "string"
+  description = "SSH public key for user 'core'"
+}
+
+variable "asset_dir" {
+  description = "Path to a directory where generated assets should be placed (contains secrets)"
+  type        = "string"
+}
+
+variable "host_cidr" {
+  description = "CIDR IPv4 range to assign to instances"
+  type        = "string"
+  default     = "10.0.0.0/16"
+}
+
+variable "pod_cidr" {
+  description = "CIDR IPv4 range to assign Kubernetes pods"
+  type        = "string"
+  default     = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+
+  type    = "string"
+  default = "10.3.0.0/16"
+}
+
+variable "cluster_domain_suffix" {
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  type        = "string"
+  default     = "cluster.local"
+}

azure/container-linux/kubernetes/workers.tf

@@ -0,0 +1,23 @@
+module "workers" {
+  source = "workers"
+  name   = "${var.cluster_name}"
+
+  # Azure
+  resource_group_name     = "${azurerm_resource_group.cluster.name}"
+  region                  = "${azurerm_resource_group.cluster.location}"
+  subnet_id               = "${azurerm_subnet.worker.id}"
+  security_group_id       = "${azurerm_network_security_group.worker.id}"
+  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+
+  count    = "${var.worker_count}"
+  vm_type  = "${var.worker_type}"
+  os_image = "${var.os_image}"
+  priority = "${var.worker_priority}"
+
+  # configuration
+  kubeconfig            = "${module.bootkube.kubeconfig}"
+  ssh_authorized_key    = "${var.ssh_authorized_key}"
+  service_cidr          = "${var.service_cidr}"
+  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  clc_snippets          = "${var.worker_clc_snippets}"
+}

azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -5,15 +5,6 @@ systemd:
       enable: true
     - name: locksmithd.service
       mask: true
-    - name: kubelet.path
-      enable: true
-      contents: |
-        [Unit]
-        Description=Watch for kubeconfig
-        [Path]
-        PathExists=/etc/kubernetes/kubeconfig
-        [Install]
-        WantedBy=multi-user.target
     - name: wait-for-dns.service
       enable: true
       contents: |
@@ -28,6 +19,7 @@ systemd:
         [Install]
         RequiredBy=kubelet.service
     - name: kubelet.service
+      enable: true
       contents: |
         [Unit]
         Description=Kubelet via Hyperkube
@@ -39,6 +31,8 @@ systemd:
           --mount volume=resolv,target=/etc/resolv.conf \
           --volume var-lib-cni,kind=host,source=/var/lib/cni \
           --mount volume=var-lib-cni,target=/var/lib/cni \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
           --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
           --mount volume=opt-cni-bin,target=/opt/cni/bin \
           --volume var-log,kind=host,source=/var/log \
@@ -47,21 +41,20 @@ systemd:
         ExecStartPre=/bin/mkdir -p /opt/cni/bin
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
         ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
         ExecStartPre=/bin/mkdir -p /var/lib/cni
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
         ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
         ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
         ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
-          --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns={{.k8s_dns_service_ip}} \
+          --cluster_dns=${k8s_dns_service_ip} \
-          --cluster_domain={{.cluster_domain_suffix}} \
+          --cluster_domain=${cluster_domain_suffix} \
           --cni-conf-dir=/etc/kubernetes/cni/net.d \
           --exit-on-lock-contention \
-          --hostname-override={{.domain_name}} \
           --kubeconfig=/etc/kubernetes/kubeconfig \
           --lock-file=/var/run/lock/kubelet.lock \
           --network-plugin=cni \
@@ -73,45 +66,56 @@ systemd:
         RestartSec=5
         [Install]
         WantedBy=multi-user.target
-
+    - name: delete-node.service
+      enable: true
+      contents: |
+        [Unit]
+        Description=Waiting to delete Kubernetes node on shutdown
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/true
+        ExecStop=/etc/kubernetes/delete-node
+        [Install]
+        WantedBy=multi-user.target
 storage:
-  {{ if index . "pxe" }}
-  disks:
-    - device: /dev/sda
-      wipe_table: true
-      partitions:
-        - label: ROOT
-  filesystems:
-    - name: root
-      mount:
-        device: "/dev/sda1"
-        format: "ext4"
-        create:
-          force: true
-          options:
-            - "-LROOT"
-  {{end}}
   files:
+    - path: /etc/kubernetes/kubeconfig
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
     - path: /etc/kubernetes/kubelet.env
       filesystem: root
       mode: 0644
       contents:
         inline: |
-          KUBELET_IMAGE_URL=docker://gcr.io/google_containers/hyperkube
+          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.9.4
+          KUBELET_IMAGE_TAG=v1.11.3
-    - path: /etc/hostname
-      filesystem: root
-      mode: 0644
-      contents:
-        inline:
-          {{.domain_name}}
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
         inline: |
           fs.inotify.max_user_watches=16184
+    - path: /etc/kubernetes/delete-node
+      filesystem: root
+      mode: 0744
+      contents:
+        inline: |
+          #!/bin/bash
+          set -e
+          exec /usr/bin/rkt run \
+            --trust-keys-from-https \
+            --volume config,kind=host,source=/etc/kubernetes \
+            --mount volume=config,target=/etc/kubernetes \
+            --insecure-options=image \
+            docker://k8s.gcr.io/hyperkube:v1.11.3 \
+            --net=host \
+            --dns=host \
+            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
 passwd:
   users:
     - name: core
       ssh_authorized_keys:
-        - {{.ssh_authorized_key}}
+        - "${ssh_authorized_key}"

azure/container-linux/kubernetes/workers/ingress.tf

@@ -0,0 +1 @@
+