Add docs fixes and Flatcar Linux announcement

* https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1103
* Update Calico from v3.1.1 to v3.1.2

CHANGES.md

@@ -4,8 +4,57 @@ Notable changes between versions.
 
 ## Latest
 
-* [Introduce](https://typhoon.psdn.io/announce/#april-26-2018) Typhoon for Fedora Atomic ([#199](https://github.com/poseidon/typhoon/pull/199))
+## v1.10.3
+
+* Kubernetes [v1.10.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1103)
+* Add [Flatcar Linux](https://docs.flatcar-linux.org/) (Container Linux derivative) as an option for AWS and bare-metal (thanks @kinvolk folks)
+* Allow bearer token authentication to the Kubelet ([#216](https://github.com/poseidon/typhoon/issues/216))
+  * Require Webhook authorization to the Kubelet
+  * Switch apiserver X509 client cert org to satisfy new authorization requirement
+* Require Terraform v0.11.x and drop support for v0.10.x ([migration guide](https://typhoon.psdn.io/topics/maintenance/#terraform-v011x))
+* Update etcd from v3.3.4 to v3.3.5 ([#213](https://github.com/poseidon/typhoon/pull/213))
+* Update Calico from v3.1.1 to v3.1.2
+
+#### AWS
+
+* Allow Flatcar Linux by setting `os_image` to flatcar-stable (default), flatcar-beta, flatcar-alpha ([#211](https://github.com/poseidon/typhoon/pull/211))
+* Replace `os_channel` variable with `os_image` to align naming across clouds
+  * Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (**action required!**)
+* Allow preemptible workers via spot instances ([#202](https://github.com/poseidon/typhoon/pull/202))
+  * Add `worker_price` to allow worker spot instances. Default to empty string for the worker autoscaling group to use regular on-demand instances
+  * Add `spot_price` to internal `workers` module for spot [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
+
+#### Bare-Metal
+
+* Allow Flatcar Linux by setting `os_channel` to flatcar-stable, flatcar-beta, flatcar-alpha ([#220](https://github.com/poseidon/typhoon/pull/220))
+* Replace `container_linux_channel` variable with `os_channel`
+  * Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (**action required!**)
+* Replace `container_linux_version` variable with `os_version`
+* Add `network_ip_autodetection_method` variable for Calico host IPv4 address detection
+  * Use Calico's default "first-found" to support single NIC and bonded NIC nodes 
+  * Allow [alternative](https://docs.projectcalico.org/v3.1/reference/node/configuration#ip-autodetection-methods) methods for multi NIC nodes, like can-reach=IP or interface=REGEX
+* Deprecate `container_linux_oem` variable
+
+#### DigitalOcean
+
+* Update Fedora Atomic module to use Fedora Atomic 28 ([#225](https://github.com/poseidon/typhoon/pull/225))
+  * Fedora Atomic 27 images disappeared from DigitalOcean and forced this early update
+
+#### Addons
+
+* Fix Prometheus data directory location ([#203](https://github.com/poseidon/typhoon/pull/203))
+* Configure Prometheus to scrape Kubelets directly with bearer token auth instead of proxying through the apiserver ([#217](https://github.com/poseidon/typhoon/pull/217))
+  * Security improvement: Drop RBAC permission from `nodes/proxy` to `nodes/metrics`
+  * Scale: Remove per-node proxied scrape load from the apiserver
+* Update Grafana from v5.04 to v5.1.3 ([#208](https://github.com/poseidon/typhoon/pull/208))
+  * Disable Grafana Google Analytics by default ([#214](https://github.com/poseidon/typhoon/issues/214))
+* Update nginx-ingress from 0.14.0 to 0.15.0
+* Annotate nginx-ingress service so Prometheus auto-discovers and scrapes service endpoints ([#222](https://github.com/poseidon/typhoon/pull/222))
+
+## v1.10.2
+
 * Kubernetes [v1.10.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#v1102)
+* [Introduce](https://typhoon.psdn.io/announce/#april-26-2018) Typhoon for Fedora Atomic ([#199](https://github.com/poseidon/typhoon/pull/199))
 * Update Calico from v3.0.4 to v3.1.1 ([#197](https://github.com/poseidon/typhoon/pull/197))
   * https://www.projectcalico.org/announcing-calico-v3-1/
   * https://github.com/projectcalico/calico/releases/tag/v3.1.0

README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/google-cloud/#preemption) (varies by platform)
@@ -30,7 +30,9 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
 | Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | beta |
-| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | very alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
+
+The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
 
 ## Documentation
 
@@ -44,7 +46,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.10.3"
   
   providers = {
     google   = "google.default"
@@ -86,9 +88,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.3
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.3
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.3
 ```
 
 List the pods.

addons/grafana/deployment.yaml

@@ -21,7 +21,7 @@ spec:
     spec:
       containers:
         - name: grafana
-          image: grafana/grafana:5.0.4
+          image: grafana/grafana:5.1.3
           env:
             - name: GF_SERVER_HTTP_PORT
               value: "8080"
@@ -31,6 +31,8 @@ spec:
               value: "true"
             - name: GF_AUTH_ANONYMOUS_ORG_ROLE
               value: Viewer
+            - name: GF_ANALYTICS_REPORTING_ENABLED
+              value: "false"
           ports:
             - name: http
               containerPort: 8080

addons/nginx-ingress/aws/deployment.yaml

@@ -23,7 +23,7 @@ spec:
       hostNetwork: true
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend
@@ -67,5 +67,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
+          securityContext:
+            runAsNonRoot: false
       restartPolicy: Always
       terminationGracePeriodSeconds: 60

addons/nginx-ingress/aws/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: nginx-ingress-controller
   namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
 spec:
   type: ClusterIP
   selector:

addons/nginx-ingress/digital-ocean/daemonset.yaml

@@ -23,7 +23,7 @@ spec:
       hostNetwork: true
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend
@@ -67,5 +67,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
+          securityContext:
+            runAsNonRoot: false
       restartPolicy: Always
       terminationGracePeriodSeconds: 60

addons/nginx-ingress/digital-ocean/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: nginx-ingress-controller
   namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
 spec:
   type: ClusterIP
   selector:

addons/nginx-ingress/google-cloud/deployment.yaml

@@ -23,7 +23,7 @@ spec:
       hostNetwork: true
       containers:
         - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
           args:
             - /nginx-ingress-controller
             - --default-backend-service=$(POD_NAMESPACE)/default-backend
@@ -67,5 +67,7 @@ spec:
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
+          securityContext:
+            runAsNonRoot: false
       restartPolicy: Always
       terminationGracePeriodSeconds: 60

addons/nginx-ingress/google-cloud/service.yaml

@@ -3,6 +3,9 @@ kind: Service
 metadata:
   name: nginx-ingress-controller
   namespace: ingress
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10254'
 spec:
   type: ClusterIP
   selector:

addons/prometheus/config.yaml

@@ -56,12 +56,7 @@ data:
         target_label: job
 
     # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
-    # metrics from a node by scraping kubelet (127.0.0.1:10255/metrics).
-    #
-    # Rather than connecting directly to the node, the scrape is proxied though the
-    # Kubernetes apiserver.  This means it will work if Prometheus is running out of
-    # cluster, or can't connect to nodes for some other reason (e.g. because of
-    # firewalling).
+    # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
     - job_name: 'kubelet'
       kubernetes_sd_configs:
       - role: node
@@ -69,50 +64,34 @@ data:
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        # Kubelet certs don't have any fixed IP SANs
+        insecure_skip_verify: true
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
-      - target_label: __address__
-        replacement: kubernetes.default.svc:443
-      - source_labels: [__meta_kubernetes_node_name]
-        regex: (.+)
-        target_label: __metrics_path__
-        replacement: /api/v1/nodes/${1}/proxy/metrics
 
     # Scrape config for Kubelet cAdvisor. Explore metrics from a node by
-    # scraping kubelet (127.0.0.1:10255/metrics/cadvisor).
-    #
-    # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
-    # (those whose names begin with 'container_') have been removed from the
-    # Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
-    # retrieve those metrics.
-    #
-    # Rather than connecting directly to the node, the scrape is proxied though the
-    # Kubernetes apiserver.  This means it will work if Prometheus is running out of
-    # cluster, or can't connect to nodes for some other reason (e.g. because of
-    # firewalling).
+    # scraping kubelet (127.0.0.1:10250/metrics/cadvisor).
     - job_name: 'kubernetes-cadvisor'
       kubernetes_sd_configs:
       - role: node
-      
+
       scheme: https
+      metrics_path: /metrics/cadvisor
       tls_config:
+        # Kubelet certs don't have any fixed IP SANs
+        insecure_skip_verify: true
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
-      - target_label: __address__
-        replacement: kubernetes.default.svc:443
-      - source_labels: [__meta_kubernetes_node_name]
-        regex: (.+)
-        target_label: __metrics_path__
-        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
-    
-    # Scrap etcd metrics from controllers 
+
+
+    # Scrap etcd metrics from controllers via listen-metrics-urls
     - job_name: 'etcd'
       kubernetes_sd_configs:
       - role: node

addons/prometheus/deployment.yaml

@@ -20,7 +20,8 @@ spec:
       - name: prometheus
         image: quay.io/prometheus/prometheus:v2.2.1
         args:
-          - '--config.file=/etc/prometheus/prometheus.yaml'
+          - --config.file=/etc/prometheus/prometheus.yaml
+          - --storage.tsdb.path=/var/lib/prometheus
         ports:
         - name: web
           containerPort: 9090

addons/prometheus/rbac/cluster-role.yaml

@@ -6,7 +6,7 @@ rules:
 - apiGroups: [""]
   resources:
   - nodes
-  - nodes/proxy
+  - nodes/metrics
   - services
   - endpoints
   - pods

aws/container-linux/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
@@ -19,5 +19,5 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/aws/).
+Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
 

aws/container-linux/kubernetes/ami.tf

@@ -1,3 +1,13 @@
+locals {
+  # Pick a CoreOS Container Linux derivative
+  # coreos-stable -> Container Linux AMI
+  # flatcar-stable -> Flatcar Linux AMI
+  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+
+  flavor  = "${element(split("-", var.os_image), 0)}"
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
 data "aws_ami" "coreos" {
   most_recent = true
   owners      = ["595879546273"]
@@ -14,6 +24,26 @@ data "aws_ami" "coreos" {
 
   filter {
     name   = "name"
-    values = ["CoreOS-${var.os_channel}-*"]
+    values = ["CoreOS-${local.channel}-*"]
+  }
+}
+
+data "aws_ami" "flatcar" {
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
   }
 }

aws/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.4"
+            Environment="ETCD_IMAGE_TAG=v3.3.5"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -76,6 +76,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -121,7 +123,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

aws/container-linux/kubernetes/controllers.tf

@@ -23,7 +23,7 @@ resource "aws_instance" "controllers" {
 
   instance_type = "${var.controller_type}"
 
-  ami       = "${data.aws_ami.coreos.image_id}"
+  ami       = "${local.ami_id}"
   user_data = "${element(data.ct_config.controller_ign.*.rendered, count.index)}"
 
   # storage

aws/container-linux/kubernetes/require.tf

@@ -1,11 +1,11 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "aws" {
-  version = "~> 1.11"
+  version = "~> 1.13"
 }
 
 provider "local" {

aws/container-linux/kubernetes/security.tf

@@ -91,6 +91,16 @@ resource "aws_security_group_rule" "controller-node-exporter" {
   source_security_group_id = "${aws_security_group.worker.id}"
 }
 
+resource "aws_security_group_rule" "controller-kubelet" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10250
+  to_port                  = 10250
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
 resource "aws_security_group_rule" "controller-kubelet-self" {
   security_group_id = "${aws_security_group.controller.id}"
 

aws/container-linux/kubernetes/variables.tf

@@ -41,10 +41,10 @@ variable "worker_type" {
   description = "EC2 instance type for workers"
 }
 
-variable "os_channel" {
+variable "os_image" {
   type        = "string"
-  default     = "stable"
-  description = "Container Linux AMI channel (stable, beta, alpha)"
+  default     = "coreos-stable"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
 }
 
 variable "disk_size" {
@@ -59,6 +59,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "worker_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
 variable "controller_clc_snippets" {
   type        = "list"
   description = "Controller Container Linux Config snippets"

aws/container-linux/kubernetes/workers.tf

@@ -8,8 +8,9 @@ module "workers" {
   security_groups = ["${aws_security_group.worker.id}"]
   count           = "${var.worker_count}"
   instance_type   = "${var.worker_type}"
-  os_channel      = "${var.os_channel}"
+  os_image        = "${var.os_image}"
   disk_size       = "${var.disk_size}"
+  spot_price      = "${var.worker_price}"
 
   # configuration
   kubeconfig            = "${module.bootkube.kubeconfig}"

aws/container-linux/kubernetes/workers/ami.tf

@@ -1,3 +1,13 @@
+locals {
+  # Pick a CoreOS Container Linux derivative
+  # coreos-stable -> Container Linux AMI
+  # flatcar-stable -> Flatcar Linux AMI
+  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+
+  flavor  = "${element(split("-", var.os_image), 0)}"
+  channel = "${element(split("-", var.os_image), 1)}"
+}
+
 data "aws_ami" "coreos" {
   most_recent = true
   owners      = ["595879546273"]
@@ -14,6 +24,26 @@ data "aws_ami" "coreos" {
 
   filter {
     name   = "name"
-    values = ["CoreOS-${var.os_channel}-*"]
+    values = ["CoreOS-${local.channel}-*"]
+  }
+}
+
+data "aws_ami" "flatcar" {
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
   }
 }

aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -49,6 +49,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -91,7 +93,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -109,7 +111,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.10.2 \
+            docker://k8s.gcr.io/hyperkube:v1.10.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

aws/container-linux/kubernetes/workers/variables.tf

@@ -34,10 +34,10 @@ variable "instance_type" {
   description = "EC2 instance type"
 }
 
-variable "os_channel" {
+variable "os_image" {
   type        = "string"
-  default     = "stable"
-  description = "Container Linux AMI channel (stable, beta, alpha)"
+  default     = "coreos-stable"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
 }
 
 variable "disk_size" {
@@ -52,6 +52,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "spot_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
 variable "clc_snippets" {
   type        = "list"
   description = "Container Linux Config snippets"

aws/container-linux/kubernetes/workers/workers.tf

@@ -26,6 +26,12 @@ resource "aws_autoscaling_group" "workers" {
     create_before_destroy = true
   }
 
+  # Waiting for instance creation delays adding the ASG to state. If instances
+  # can't be created (e.g. spot price too low), the ASG will be orphaned.
+  # Orphaned ASGs escape cleanup, can't be updated, and keep bidding if spot is
+  # used. Disable wait to avoid issues and align with other clouds.
+  wait_for_capacity_timeout = "0"
+
   tags = [{
     key                 = "Name"
     value               = "${var.name}-worker"
@@ -35,8 +41,9 @@ resource "aws_autoscaling_group" "workers" {
 
 # Worker template
 resource "aws_launch_configuration" "worker" {
-  image_id      = "${data.aws_ami.coreos.image_id}"
+  image_id      = "${local.ami_id}"
   instance_type = "${var.instance_type}"
+  spot_price    = "${var.spot_price}"
 
   user_data = "${data.ct_config.worker_ign.rendered}"
 

aws/fedora-atomic/kubernetes/README.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/)
@@ -19,5 +19,5 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/aws/).
+Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
 

aws/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -53,6 +53,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -91,8 +93,8 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.4"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.5"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.12.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, cloud-metadata.service]

aws/fedora-atomic/kubernetes/require.tf

@@ -1,11 +1,11 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "aws" {
-  version = "~> 1.11"
+  version = "~> 1.13"
 }
 
 provider "local" {

aws/fedora-atomic/kubernetes/security.tf

@@ -91,6 +91,16 @@ resource "aws_security_group_rule" "controller-node-exporter" {
   source_security_group_id = "${aws_security_group.worker.id}"
 }
 
+resource "aws_security_group_rule" "controller-kubelet" {
+  security_group_id = "${aws_security_group.controller.id}"
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10250
+  to_port                  = 10250
+  source_security_group_id = "${aws_security_group.worker.id}"
+}
+
 resource "aws_security_group_rule" "controller-kubelet-self" {
   security_group_id = "${aws_security_group.controller.id}"
 

aws/fedora-atomic/kubernetes/variables.tf

@@ -53,6 +53,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "worker_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
 # configuration
 
 variable "ssh_authorized_key" {

aws/fedora-atomic/kubernetes/workers.tf

@@ -9,6 +9,7 @@ module "workers" {
   count           = "${var.worker_count}"
   instance_type   = "${var.worker_type}"
   disk_size       = "${var.disk_size}"
+  spot_price      = "${var.worker_price}"
 
   # configuration
   kubeconfig            = "${module.bootkube.kubeconfig}"

aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -32,6 +32,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -68,7 +70,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default

aws/fedora-atomic/kubernetes/workers/variables.tf

@@ -46,6 +46,12 @@ variable "disk_type" {
   description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }
 
+variable "spot_price" {
+  type        = "string"
+  default     = ""
+  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
+}
+
 # configuration
 
 variable "kubeconfig" {

aws/fedora-atomic/kubernetes/workers/workers.tf

@@ -26,6 +26,12 @@ resource "aws_autoscaling_group" "workers" {
     create_before_destroy = true
   }
 
+  # Waiting for instance creation delays adding the ASG to state. If instances
+  # can't be created (e.g. spot price too low), the ASG will be orphaned.
+  # Orphaned ASGs escape cleanup, can't be updated, and keep bidding if spot is
+  # used. Disable wait to avoid issues and align with other clouds.
+  wait_for_capacity_timeout = "0"
+
   tags = [{
     key                 = "Name"
     value               = "${var.name}-worker"
@@ -37,6 +43,7 @@ resource "aws_autoscaling_group" "workers" {
 resource "aws_launch_configuration" "worker" {
   image_id      = "${data.aws_ami.fedora.image_id}"
   instance_type = "${var.instance_type}"
+  spot_price    = "${var.spot_price}"
 
   user_data = "${data.template_file.worker-cloudinit.rendered}"
 

bare-metal/container-linux/kubernetes/README.md

@@ -11,12 +11,12 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/bare-metal/).
+Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/cl/bare-metal/).
 

bare-metal/container-linux/kubernetes/bootkube.tf

@@ -1,14 +1,15 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
-  cluster_name          = "${var.cluster_name}"
-  api_servers           = ["${var.k8s_domain_name}"]
-  etcd_servers          = ["${var.controller_domains}"]
-  asset_dir             = "${var.asset_dir}"
-  networking            = "${var.networking}"
-  network_mtu           = "${var.network_mtu}"
-  pod_cidr              = "${var.pod_cidr}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
+  cluster_name                    = "${var.cluster_name}"
+  api_servers                     = ["${var.k8s_domain_name}"]
+  etcd_servers                    = ["${var.controller_domains}"]
+  asset_dir                       = "${var.asset_dir}"
+  networking                      = "${var.networking}"
+  network_mtu                     = "${var.network_mtu}"
+  network_ip_autodetection_method = "${var.network_ip_autodetection_method}"
+  pod_cidr                        = "${var.pod_cidr}"
+  service_cidr                    = "${var.service_cidr}"
+  cluster_domain_suffix           = "${var.cluster_domain_suffix}"
 }

bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.4"
+            Environment="ETCD_IMAGE_TAG=v3.3.5"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@@ -84,6 +84,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -122,7 +124,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/cl/install.yaml.tmpl

@@ -31,10 +31,10 @@ storage:
         inline: |
           #!/bin/bash -ex
           curl --retry 10 "${ignition_endpoint}?{{.request.raw_query}}&os=installed" -o ignition.json
-          coreos-install \
+          ${os_flavor}-install \
             -d ${install_disk} \
-            -C ${container_linux_channel} \
-            -V ${container_linux_version} \
+            -C ${os_channel} \
+            -V ${os_version} \
             -o "${container_linux_oem}" \
             ${baseurl_flag} \
             -i ignition.json

bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -57,6 +57,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -83,7 +85,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/hostname
       filesystem: root
       mode: 0644

bare-metal/container-linux/kubernetes/groups.tf

@@ -1,9 +1,9 @@
-// Install Container Linux to disk
-resource "matchbox_group" "container-linux-install" {
+resource "matchbox_group" "install" {
   count = "${length(var.controller_names) + length(var.worker_names)}"
 
-  name    = "${format("container-linux-install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
-  profile = "${var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
+  name    = "${format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
+
+  profile = "${local.flavor == "flatcar" ? element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
 
   selector {
     mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"

bare-metal/container-linux/kubernetes/profiles.tf

@@ -1,12 +1,19 @@
+locals {
+  # coreos-stable -> coreos flavor, stable channel
+  # flatcar-stable -> flatcar flavor, stable channel
+  flavor  = "${element(split("-", var.os_channel), 0)}"
+  channel = "${element(split("-", var.os_channel), 1)}" 
+}
+
 // Container Linux Install profile (from release.core-os.net)
 resource "matchbox_profile" "container-linux-install" {
   count = "${length(var.controller_names) + length(var.worker_names)}"
   name  = "${format("%s-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
 
-  kernel = "http://${var.container_linux_channel}.release.core-os.net/amd64-usr/${var.container_linux_version}/coreos_production_pxe.vmlinuz"
+  kernel = "http://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe.vmlinuz"
 
   initrd = [
-    "http://${var.container_linux_channel}.release.core-os.net/amd64-usr/${var.container_linux_version}/coreos_production_pxe_image.cpio.gz",
+    "http://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe_image.cpio.gz",
   ]
 
   args = [
@@ -24,15 +31,16 @@ resource "matchbox_profile" "container-linux-install" {
 data "template_file" "container-linux-install-configs" {
   count = "${length(var.controller_names) + length(var.worker_names)}"
 
-  template = "${file("${path.module}/cl/container-linux-install.yaml.tmpl")}"
+  template = "${file("${path.module}/cl/install.yaml.tmpl")}"
 
   vars {
-    container_linux_channel = "${var.container_linux_channel}"
-    container_linux_version = "${var.container_linux_version}"
-    ignition_endpoint       = "${format("%s/ignition", var.matchbox_http_endpoint)}"
-    install_disk            = "${var.install_disk}"
-    container_linux_oem     = "${var.container_linux_oem}"
-    ssh_authorized_key      = "${var.ssh_authorized_key}"
+    os_flavor           = "${local.flavor}"
+    os_channel          = "${local.channel}"
+    os_version          = "${var.os_version}"
+    ignition_endpoint   = "${format("%s/ignition", var.matchbox_http_endpoint)}"
+    install_disk        = "${var.install_disk}"
+    container_linux_oem = "${var.container_linux_oem}"
+    ssh_authorized_key  = "${var.ssh_authorized_key}"
 
     # only cached-container-linux profile adds -b baseurl
     baseurl_flag = ""
@@ -40,15 +48,15 @@ data "template_file" "container-linux-install-configs" {
 }
 
 // Container Linux Install profile (from matchbox /assets cache)
-// Note: Admin must have downloaded container_linux_version into matchbox assets.
+// Note: Admin must have downloaded os_version into matchbox assets.
 resource "matchbox_profile" "cached-container-linux-install" {
   count = "${length(var.controller_names) + length(var.worker_names)}"
   name  = "${format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
 
-  kernel = "/assets/coreos/${var.container_linux_version}/coreos_production_pxe.vmlinuz"
+  kernel = "/assets/coreos/${var.os_version}/coreos_production_pxe.vmlinuz"
 
   initrd = [
-    "/assets/coreos/${var.container_linux_version}/coreos_production_pxe_image.cpio.gz",
+    "/assets/coreos/${var.os_version}/coreos_production_pxe_image.cpio.gz",
   ]
 
   args = [
@@ -66,21 +74,45 @@ resource "matchbox_profile" "cached-container-linux-install" {
 data "template_file" "cached-container-linux-install-configs" {
   count = "${length(var.controller_names) + length(var.worker_names)}"
 
-  template = "${file("${path.module}/cl/container-linux-install.yaml.tmpl")}"
+  template = "${file("${path.module}/cl/install.yaml.tmpl")}"
 
   vars {
-    container_linux_channel = "${var.container_linux_channel}"
-    container_linux_version = "${var.container_linux_version}"
-    ignition_endpoint       = "${format("%s/ignition", var.matchbox_http_endpoint)}"
-    install_disk            = "${var.install_disk}"
-    container_linux_oem     = "${var.container_linux_oem}"
-    ssh_authorized_key      = "${var.ssh_authorized_key}"
+    os_flavor           = "${local.flavor}"
+    os_channel          = "${local.channel}"
+    os_version          = "${var.os_version}"
+    ignition_endpoint   = "${format("%s/ignition", var.matchbox_http_endpoint)}"
+    install_disk        = "${var.install_disk}"
+    container_linux_oem = "${var.container_linux_oem}"
+    ssh_authorized_key  = "${var.ssh_authorized_key}"
 
     # profile uses -b baseurl to install from matchbox cache
     baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/coreos"
   }
 }
 
+// Flatcar Linux install profile (from release.flatcar-linux.net)
+resource "matchbox_profile" "flatcar-install" {
+  count = "${length(var.controller_names) + length(var.worker_names)}"
+  name  = "${format("%s-flatcar-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+
+  kernel = "http://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe.vmlinuz"
+
+  initrd = [
+    "http://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
+  ]
+
+  args = [
+    "initrd=flatcar_production_pxe_image.cpio.gz",
+    "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
+    "flatcar.first_boot=yes",
+    "console=tty0",
+    "console=ttyS0",
+    "${var.kernel_args}",
+  ]
+
+  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
+}
+
 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
   count                  = "${length(var.controller_names)}"

bare-metal/container-linux/kubernetes/require.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "local" {

bare-metal/container-linux/kubernetes/variables.tf

@@ -10,14 +10,14 @@ variable "matchbox_http_endpoint" {
   description = "Matchbox HTTP read-only endpoint (e.g. http://matchbox.example.com:8080)"
 }
 
-variable "container_linux_channel" {
+variable "os_channel" {
   type        = "string"
-  description = "Container Linux channel corresponding to the container_linux_version"
+  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
 }
 
-variable "container_linux_version" {
+variable "os_version" {
   type        = "string"
-  description = "Container Linux version of the kernel/initrd to PXE or the image to install"
+  description = "Version for a Container Linux derivative to PXE and install (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
 }
 
 # machines
@@ -76,6 +76,12 @@ variable "network_mtu" {
   default     = "1480"
 }
 
+variable "network_ip_autodetection_method" {
+  description = "Method to autodetect the host IPv4 address (applies to calico only)"
+  type        = "string"
+  default     = "first-found"
+}
+
 variable "pod_cidr" {
   description = "CIDR IPv4 range to assign Kubernetes pods"
   type        = "string"
@@ -103,7 +109,7 @@ variable "cluster_domain_suffix" {
 variable "cached_install" {
   type        = "string"
   default     = "false"
-  description = "Whether Container Linux should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the container_linux_version into matchbox assets."
+  description = "Whether Container Linux should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the os_version into matchbox assets."
 }
 
 variable "install_disk" {
@@ -115,7 +121,7 @@ variable "install_disk" {
 variable "container_linux_oem" {
   type        = "string"
   default     = ""
-  description = "Specify an OEM image id to use as base for the installation (e.g. ami, vmware_raw, xen) or leave blank for the default image"
+  description = "DEPRECATED: Specify an OEM image id to use as base for the installation (e.g. ami, vmware_raw, xen) or leave blank for the default image"
 }
 
 variable "kernel_args" {

bare-metal/fedora-atomic/kubernetes/README.md

@@ -11,12 +11,12 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/bare-metal/).
+Please see the [official docs](https://typhoon.psdn.io) and the bare-metal [tutorial](https://typhoon.psdn.io/cl/bare-metal/).
 

bare-metal/fedora-atomic/kubernetes/bootkube.tf

@@ -1,7 +1,7 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
-  
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
+
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${var.k8s_domain_name}"]
   etcd_servers          = ["${var.controller_domains}"]

bare-metal/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -38,6 +38,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -82,8 +84,8 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.4"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.5"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.12.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, kubelet.path]

bare-metal/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -17,6 +17,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -58,7 +60,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [hostnamectl, set-hostname, ${domain_name}]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

bare-metal/fedora-atomic/kubernetes/profiles.tf

@@ -1,6 +1,6 @@
 locals {
   default_assets_endpoint = "${var.matchbox_http_endpoint}/assets/fedora/27"
-  atomic_assets_endpoint = "${var.atomic_assets_endpoint != "" ? var.atomic_assets_endpoint : local.default_assets_endpoint}"
+  atomic_assets_endpoint  = "${var.atomic_assets_endpoint != "" ? var.atomic_assets_endpoint : local.default_assets_endpoint}"
 }
 
 // Cached Fedora Install profile (from matchbox /assets cache)
@@ -36,14 +36,15 @@ data "template_file" "install-kickstarts" {
   vars {
     matchbox_http_endpoint = "${var.matchbox_http_endpoint}"
     atomic_assets_endpoint = "${local.atomic_assets_endpoint}"
-    mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
+    mac                    = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
   }
 }
 
 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
-  count                  = "${length(var.controller_names)}"
-  name                   = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
+  count = "${length(var.controller_names)}"
+  name  = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
+
   # cloud-init
   generic_config = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
 }
@@ -65,8 +66,9 @@ data "template_file" "controller-configs" {
 
 // Kubernetes Worker profiles
 resource "matchbox_profile" "workers" {
-  count                  = "${length(var.worker_names)}"
-  name                   = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
+  count = "${length(var.worker_names)}"
+  name  = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
+
   # cloud-init
   generic_config = "${element(data.template_file.worker-configs.*.rendered, count.index)}"
 }

bare-metal/fedora-atomic/kubernetes/require.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "local" {

bare-metal/fedora-atomic/kubernetes/variables.tf

@@ -11,8 +11,9 @@ variable "matchbox_http_endpoint" {
 }
 
 variable "atomic_assets_endpoint" {
-  type = "string"
+  type    = "string"
   default = ""
+
   description = <<EOD
 HTTP endpoint serving the Fedora Atomic Host vmlinuz, initrd, os repo, and ostree repo (.e.g `http://example.com/some/path`).
 

digital-ocean/container-linux/kubernetes/README.md

@@ -11,12 +11,12 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the Digital Ocean [tutorial](https://typhoon.psdn.io/digital-ocean/).
+Please see the [official docs](https://typhoon.psdn.io) and the Digital Ocean [tutorial](https://typhoon.psdn.io/cl/digital-ocean/).
 

digital-ocean/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.4"
+            Environment="ETCD_IMAGE_TAG=v3.3.5"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -87,6 +87,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -127,7 +129,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl

@@ -60,6 +60,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -97,7 +99,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -115,7 +117,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.10.2 \
+            docker://k8s.gcr.io/hyperkube:v1.10.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

digital-ocean/container-linux/kubernetes/require.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "digitalocean" {

digital-ocean/fedora-atomic/kubernetes/README.md

@@ -11,12 +11,12 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the Digital Ocean [tutorial](https://typhoon.psdn.io/digital-ocean/).
+Please see the [official docs](https://typhoon.psdn.io) and the Digital Ocean [tutorial](https://typhoon.psdn.io/cl/digital-ocean/).
 

digital-ocean/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

digital-ocean/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -53,6 +53,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -88,8 +90,8 @@ bootcmd:
   - [modprobe, ip_vs]
 runcmd:
   - [systemctl, daemon-reload]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.4"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.5"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.12.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, cloud-metadata.service]

digital-ocean/fedora-atomic/kubernetes/cloudinit/worker.yaml.tmpl

@@ -32,6 +32,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -65,7 +67,7 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - [systemctl, enable, kubelet.path]
   - [systemctl, start, --no-block, kubelet.path]
 users:

digital-ocean/fedora-atomic/kubernetes/controllers.tf

@@ -46,7 +46,7 @@ resource "digitalocean_droplet" "controllers" {
 
   user_data = "${element(data.template_file.controller-cloudinit.*.rendered, count.index)}"
   ssh_keys  = ["${var.ssh_fingerprints}"]
-  
+
   tags = [
     "${digitalocean_tag.controllers.id}",
   ]

digital-ocean/fedora-atomic/kubernetes/require.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "digitalocean" {

digital-ocean/fedora-atomic/kubernetes/variables.tf

@@ -43,8 +43,8 @@ variable "worker_type" {
 
 variable "image" {
   type        = "string"
-  default     = "fedora-27-x64-atomic"
-  description = "OS image from which to initialize the disk (e.g. fedora-27-x64-atomic)"
+  default     = "fedora-28-x64-atomic"
+  description = "OS image from which to initialize the disk (e.g. fedora-28-x64-atomic)"
 }
 
 # configuration

docs/advanced/worker-pools.md

@@ -15,7 +15,7 @@ Create a cluster following the AWS [tutorial](../cl/aws.md#cluster). Define a wo
 
 ```tf
 module "tempest-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes/workers?ref=v1.10.3"
   
   providers = {
     aws = "aws.default"
@@ -33,7 +33,7 @@ module "tempest-worker-pool" {
 
   count         = 2
   instance_type = "m5.large"
-  os_channel    = "beta"    
+  os_image      = "coreos-beta"
 }
 ```
 
@@ -66,12 +66,13 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 |:-----|:------------|:--------|:--------|
 | count | Number of instances | 1 | 3 |
 | instance_type | EC2 instance type | "t2.small" | "t2.medium" |
-| os_channel | Container Linux AMI channel | stable| "beta", "alpha" |
+| os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the disk in GB | 40 | 100 |
+| spot_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | cluster_domain_suffix | Must match `cluster_domain_suffix` of cluster | "cluster.local" | "k8s.example.com" |
 
-Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/).
+Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-types/) or per-region and per-type [spot prices](https://aws.amazon.com/ec2/spot/pricing/).
 
 ## Google Cloud
 
@@ -79,7 +80,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu
 
 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.10.3"
 
   providers = {
     google = "google.default"
@@ -113,11 +114,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.10.2
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.10.2
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.10.2
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.10.2
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.10.2
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.10.3
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.10.3
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.10.3
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.10.3
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.10.3
 ```
 
 ### Variables

docs/announce.md

@@ -1,5 +1,15 @@
 # Announce <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo-small.png">
 
+## May 23, 2018
+
+Starting in v1.10.3, Typhoon AWS and bare-metal `container-linux` modules allow picking between the Red Hat [Container Linux](https://coreos.com/os/docs/latest/) (formerly CoreOS Container Linux) and Kinvolk [Flatcar Linux](https://www.flatcar-linux.org/) operating system. Flatcar Linux serves as a drop-in compatible "friendly fork" of Container Linux. Flatcar Linux publishes the same channels and versions as Container Linux and gets provisioned, managed, and operated in an identical way (e.g. login as user "core").
+
+On AWS, pick the Container Linux derivative channel by setting `os_image` to coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, or flatcar-alpha.
+
+On bare-metal, pick the Container Linux derivative channel by setting `os_channel` to coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, or flatcar-alpha. Set the `os_version` number to PXE boot and install. Variables `container_linux_channel` and `container_linux_version` have been dropped.
+
+Flatcar Linux provides a familar Container Linux experience, with support from Kinvolk as an alternative to Red Hat. Typhoon offers the choice of Container Linux vendor to satisfy differing preferences and to diversify technology underpinnings, while providing a consistent Kubernetes experience across operating systems, clouds, and on-premise.
+
 ## April 26, 2018
 
 Introducing Typhoon Kubernetes clusters for Fedora Atomic!

docs/atomic/aws.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.10.2 cluster on AWS with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.10.3 cluster on AWS with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -44,7 +44,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf
 
 ```tf
 provider "aws" {
-  version = "~> 1.11.0"
+  version = "~> 1.13.0"
   alias   = "default"
 
   region                  = "eu-central-1"
@@ -83,7 +83,7 @@ Define a Kubernetes cluster using the module `aws/fedora-atomic/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-atomic/kubernetes?ref=v1.10.3"
 
   providers = {
     aws = "aws.default"
@@ -156,9 +156,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.10.2
-ip-10-0-19-112   Ready     34m       v1.10.2
-ip-10-0-4-22     Ready     34m       v1.10.2
+ip-10-0-12-221   Ready     34m       v1.10.3
+ip-10-0-19-112   Ready     34m       v1.10.3
+ip-10-0-4-22     Ready     34m       v1.10.3
 ```
 
 List the pods.
@@ -227,6 +227,7 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | worker_type | EC2 instance type for workers | "t2.small" | See below |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
 | host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |
@@ -238,4 +239,3 @@ Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-typ
 
 !!! tip "MTU"
     If your EC2 instance type supports [Jumbo frames](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) (most do), we recommend you change the `network_mtu` to 8991! You will get better pod-to-pod bandwidth.
-

docs/atomic/bare-metal.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.10.2 cluster on bare-metal with Fedora Atomic.
+In this tutorial, we'll network boot and provision a Kubernetes v1.10.3 cluster on bare-metal with Fedora Atomic.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora Atomic via kickstart, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via cloud-init.
 
@@ -234,7 +234,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-atomic/kubernete
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-atomic/kubernetes?ref=v1.10.3"
   
   providers = {
     local = "local.default"
@@ -360,9 +360,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.10.2
-node2.example.com   Ready     11m       v1.10.2
-node3.example.com   Ready     11m       v1.10.2
+node1.example.com   Ready     11m       v1.10.3
+node2.example.com   Ready     11m       v1.10.3
+node3.example.com   Ready     11m       v1.10.3
 ```
 
 List the pods.

docs/atomic/digital-ocean.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is alpha. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.10.2 cluster on DigitalOcean with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.10.3 cluster on DigitalOcean with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -77,7 +77,7 @@ Define a Kubernetes cluster using the module `digital-ocean/fedora-atomic/kubern
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-atomic/kubernetes?ref=v1.10.3"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -152,9 +152,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.10.2
-10.132.115.81    Ready     10m       v1.10.2
-10.132.124.107   Ready     10m       v1.10.2
+10.132.110.130   Ready     10m       v1.10.3
+10.132.115.81    Ready     10m       v1.10.3
+10.132.124.107   Ready     10m       v1.10.3
 ```
 
 List the pods.

docs/atomic/google-cloud.md

@@ -3,7 +3,7 @@
 !!! danger
     Typhoon for Fedora Atomic is very alpha. Fedora does not publish official images for Google Cloud so you must prepare them yourself. Some addons don't work yet. Expect rough edges and changes.
 
-In this tutorial, we'll create a Kubernetes v1.10.2 cluster on Google Compute Engine with Fedora Atomic.
+In this tutorial, we'll create a Kubernetes v1.10.3 cluster on Google Compute Engine with Fedora Atomic.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets. Instances are provisioned on first boot with cloud-init.
 
@@ -119,7 +119,7 @@ Define a Kubernetes cluster using the module `google-cloud/fedora-atomic/kuberne
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-atomic/kubernetes?ref=v1.10.3"
   
   providers = {
     google   = "google.default"
@@ -195,9 +195,9 @@ In 5-10 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.3
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.3
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.3
 ```
 
 List the pods.

docs/cl/aws.md

@@ -1,6 +1,6 @@
 # AWS
 
-In this tutorial, we'll create a Kubernetes v1.10.2 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.10.3 cluster on AWS with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancers, and TLS assets.
 
@@ -57,7 +57,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf
 
 ```tf
 provider "aws" {
-  version = "~> 1.11.0"
+  version = "~> 1.13.0"
   alias   = "default"
 
   region                  = "eu-central-1"
@@ -96,7 +96,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.
 
 ```tf
 module "aws-tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.10.3"
 
   providers = {
     aws = "aws.default"
@@ -169,9 +169,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/tempest/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION        
-ip-10-0-12-221   Ready     34m       v1.10.2
-ip-10-0-19-112   Ready     34m       v1.10.2
-ip-10-0-4-22     Ready     34m       v1.10.2
+ip-10-0-12-221   Ready     34m       v1.10.3
+ip-10-0-19-112   Ready     34m       v1.10.3
+ip-10-0-4-22     Ready     34m       v1.10.3
 ```
 
 List the pods.
@@ -241,9 +241,10 @@ Reference the DNS zone id with `"${aws_route53_zone.zone-for-clusters.zone_id}"`
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | EC2 instance type for controllers | "t2.small" | See below |
 | worker_type | EC2 instance type for workers | "t2.small" | See below |
-| os_channel | Container Linux AMI channel | stable | stable, beta, alpha |
+| os_image | AMI channel for a Container Linux derivative | coreos-stable | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
 | disk_size | Size of the EBS volume in GB | "40" | "100" |
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
+| worker_price | Spot price in USD for workers. Leave as default empty string for regular on-demand instances | "" | "0.10" |
 | controller_clc_snippets | Controller Container Linux Config snippets | [] | |
 | worker_clc_snippets | Worker Container Linux Config snippets | [] | |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
@@ -257,4 +258,3 @@ Check the list of valid [instance types](https://aws.amazon.com/ec2/instance-typ
 
 !!! tip "MTU"
     If your EC2 instance type supports [Jumbo frames](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances) (most do), we recommend you change the `network_mtu` to 8991! You will get better pod-to-pod bandwidth.
-

docs/cl/bare-metal.md

@@ -1,6 +1,6 @@
 # Bare-Metal
 
-In this tutorial, we'll network boot and provision a Kubernetes v1.10.2 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.10.3 cluster on bare-metal with Container Linux.
 
 First, we'll deploy a [Matchbox](https://github.com/coreos/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.
 
@@ -174,7 +174,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne
 
 ```tf
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.10.3"
   
   providers = {
     local = "local.default"
@@ -186,8 +186,8 @@ module "bare-metal-mercury" {
   # bare-metal
   cluster_name            = "mercury"
   matchbox_http_endpoint  = "http://matchbox.example.com"
-  container_linux_channel = "stable"
-  container_linux_version = "1632.3.0"
+  os_channel              = "coreos-stable"
+  os_version              = "1632.3.0"
 
   # configuration
   k8s_domain_name    = "node1.example.com"
@@ -283,9 +283,9 @@ Apply complete! Resources: 55 added, 0 changed, 0 destroyed.
 To watch the install to disk (until machines reboot from disk), SSH to port 2222.
 
 ```
-# before v1.10.2
+# before v1.10.3
 $ ssh debug@node1.example.com
-# after v1.10.2
+# after v1.10.3
 $ ssh -p 2222 core@node1.example.com
 ```
 
@@ -310,9 +310,9 @@ bootkube[5]: Tearing down temporary bootstrap control plane...
 $ export KUBECONFIG=/home/user/.secrets/clusters/mercury/auth/kubeconfig
 $ kubectl get nodes
 NAME                STATUS    AGE       VERSION
-node1.example.com   Ready     11m       v1.10.2
-node2.example.com   Ready     11m       v1.10.2
-node3.example.com   Ready     11m       v1.10.2
+node1.example.com   Ready     11m       v1.10.3
+node2.example.com   Ready     11m       v1.10.3
+node3.example.com   Ready     11m       v1.10.3
 ```
 
 List the pods.
@@ -353,8 +353,8 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 |:-----|:------------|:--------|
 | cluster_name | Unique cluster name | mercury |
 | matchbox_http_endpoint | Matchbox HTTP read-only endpoint | http://matchbox.example.com:port |
-| container_linux_channel | Container Linux channel | stable, beta, alpha |
-| container_linux_version | Container Linux version of the kernel/initrd to PXE and the image to install | 1632.3.0 |
+| os_channel | Channel for a Container Linux derivative | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha |
+| os_version | Version for a Container Linux derivative to PXE and install | 1632.3.0 |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3Nz..." |
 | asset_dir | Path to a directory where generated assets should be placed (contains secrets) | "/home/user/.secrets/clusters/mercury" |
@@ -369,11 +369,11 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 
 | Name | Description | Default | Example |
 |:-----|:------------|:--------|:--------|
-| cached_install | Whether machines should PXE boot and install from the Matchbox `/assets` cache. Admin MUST have downloaded Container Linux images into the cache to use this | false | true |
+| cached_install | Whether machines should PXE boot and install from the Matchbox `/assets` cache. Admin MUST have downloaded Container Linux images into the cache to use this (coreos only for now) | false | true |
 | install_disk | Disk device where Container Linux should be installed | "/dev/sda" | "/dev/sdb" |
-| container_linux_oem | Specify alternative OEM image ids for the disk install | "" | "vmware_raw", "xen" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico-only) | 1480 | - | 
+| network_ip_autodetection_method | Method to detect host IPv4 address (calico-only) | first-found | can-reach=10.0.0.1 |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | cluster_domain_suffix | FQDN suffix for Kubernetes services answered by kube-dns. | "cluster.local" | "k8s.example.com" |

docs/cl/digital-ocean.md

@@ -1,6 +1,6 @@
 # Digital Ocean
 
-In this tutorial, we'll create a Kubernetes v1.10.2 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.10.3 cluster on DigitalOcean with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
 
@@ -90,7 +90,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube
 
 ```tf
 module "digital-ocean-nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.10.3"
   
   providers = {
     digitalocean = "digitalocean.default"
@@ -164,9 +164,9 @@ In 3-6 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/nemo/auth/kubeconfig
 $ kubectl get nodes
 NAME             STATUS    AGE       VERSION
-10.132.110.130   Ready     10m       v1.10.2
-10.132.115.81    Ready     10m       v1.10.2
-10.132.124.107   Ready     10m       v1.10.2
+10.132.110.130   Ready     10m       v1.10.3
+10.132.115.81    Ready     10m       v1.10.3
+10.132.124.107   Ready     10m       v1.10.3
 ```
 
 List the pods.

docs/cl/google-cloud.md

@@ -1,6 +1,6 @@
 # Google Cloud
 
-In this tutorial, we'll create a Kubernetes v1.10.2 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.10.3 cluster on Google Compute Engine with Container Linux.
 
 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
 
@@ -97,7 +97,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.10.3"
   
   providers = {
     google   = "google.default"
@@ -172,9 +172,9 @@ In 4-8 minutes, the Kubernetes cluster will be ready.
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.3
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.3
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.3
 ```
 
 List the pods.

docs/index.md

@@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [preemption](https://typhoon.psdn.io/google-cloud/#preemption) (varies by platform)
@@ -30,7 +30,9 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
 | Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](atomic/digital-ocean.md) | alpha |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | beta |
-| Google Cloud  | Fedora Atomic    | [google-cloud/container-linux/kubernetes](atomic/google-cloud.md) | very alpha |
+| Google Cloud  | Fedora Atomic    | [google-cloud/container-linux/kubernetes](atomic/google-cloud.md) | alpha |
+
+The AWS and bare-metal `container-linux` modules allow picking Red Hat Container Linux (formerly CoreOS Container Linux) or Kinvolk's Flatcar Linux friendly fork.
 
 ## Documentation
 
@@ -43,7 +45,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo
 
 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.10.3"
   
   providers = {
     google   = "google.default"
@@ -84,9 +86,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                          STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.2
-yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.2
-yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.2
+yavin-controller-0.c.example-com.internal     Ready    6m     v1.10.3
+yavin-worker-jrbf.c.example-com.internal      Ready    5m     v1.10.3
+yavin-worker-mzdm.c.example-com.internal      Ready    5m     v1.10.3
 ```
 
 List the pods.

docs/topics/maintenance.md

@@ -18,7 +18,7 @@ module "google-cloud-yavin" {
 }
 
 module "bare-metal-mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.10.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.10.3"
   ...
 }
 ```

docs/topics/performance.md

@@ -9,7 +9,7 @@ Provisioning times vary based on the operating system and platform. Sampling the
 | AWS           | 6 min | 5 min   |
 | Bare-Metal    | 10-15 min | NA  |
 | Digital Ocean | 3 min 30 sec | 20 sec |
-| Google Cloud  | 9 min | 4 min 30 sec |
+| Google Cloud  | 10 min | 4 min 30 sec |
 
 Notes:
 

google-cloud/container-linux/kubernetes/README.md

@@ -11,12 +11,12 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
 
-* Kubernetes v1.10.2 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.10.3 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, workloads isolated on workers, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/google-cloud/).
+Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/cl/google-cloud/).
 

google-cloud/container-linux/kubernetes/apiserver.tf

@@ -14,43 +14,45 @@ resource "google_dns_record_set" "apiserver" {
 
 # Static IPv4 address for the TCP Proxy Load Balancer
 resource "google_compute_global_address" "apiserver-ipv4" {
-  name = "${var.cluster_name}-apiserver-ip"
+  name       = "${var.cluster_name}-apiserver-ip"
   ip_version = "IPV4"
 }
 
 # Forward IPv4 TCP traffic to the TCP proxy load balancer
 resource "google_compute_global_forwarding_rule" "apiserver" {
-  name = "${var.cluster_name}-apiserver"
-  ip_address = "${google_compute_global_address.apiserver-ipv4.address}"
+  name        = "${var.cluster_name}-apiserver"
+  ip_address  = "${google_compute_global_address.apiserver-ipv4.address}"
   ip_protocol = "TCP"
-  port_range = "443"
-  target = "${google_compute_target_tcp_proxy.apiserver.self_link}"
+  port_range  = "443"
+  target      = "${google_compute_target_tcp_proxy.apiserver.self_link}"
 }
 
 # Global TCP Proxy Load Balancer for apiservers
 resource "google_compute_target_tcp_proxy" "apiserver" {
-  name = "${var.cluster_name}-apiserver"
-  description = "Distribute TCP load across ${var.cluster_name} controllers"
+  name            = "${var.cluster_name}-apiserver"
+  description     = "Distribute TCP load across ${var.cluster_name} controllers"
   backend_service = "${google_compute_backend_service.apiserver.self_link}"
 }
 
 # Global backend service backed by unmanaged instance groups
 resource "google_compute_backend_service" "apiserver" {
-  name = "${var.cluster_name}-apiserver"
+  name        = "${var.cluster_name}-apiserver"
   description = "${var.cluster_name} apiserver service"
 
-  protocol = "TCP"
-  port_name = "apiserver"
+  protocol         = "TCP"
+  port_name        = "apiserver"
   session_affinity = "NONE"
-  timeout_sec = "60"
+  timeout_sec      = "60"
 
   # controller(s) spread across zonal instance groups
   backend {
     group = "${google_compute_instance_group.controllers.0.self_link}"
   }
+
   backend {
     group = "${google_compute_instance_group.controllers.1.self_link}"
   }
+
   backend {
     group = "${google_compute_instance_group.controllers.2.self_link}"
   }
@@ -74,22 +76,22 @@ resource "google_compute_instance_group" "controllers" {
   instances = [
     "${matchkeys(google_compute_instance.controllers.*.self_link,
       google_compute_instance.controllers.*.zone,
-      list(element(local.zones, count.index)))}"
+      list(element(local.zones, count.index)))}",
   ]
 }
 
 # TCP health check for apiserver
 resource "google_compute_health_check" "apiserver" {
-  name = "${var.cluster_name}-apiserver-tcp-health"
+  name        = "${var.cluster_name}-apiserver-tcp-health"
   description = "TCP health check for kube-apiserver"
 
-  timeout_sec = 5
+  timeout_sec        = 5
   check_interval_sec = 5
 
-  healthy_threshold = 1
+  healthy_threshold   = 1
   unhealthy_threshold = 3
 
   tcp_health_check {
-    port  = "443"
+    port = "443"
   }
 }

google-cloud/container-linux/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl

@@ -7,7 +7,7 @@ systemd:
         - name: 40-etcd-cluster.conf
           contents: |
             [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.4"
+            Environment="ETCD_IMAGE_TAG=v3.3.5"
             Environment="ETCD_NAME=${etcd_name}"
             Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
             Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@@ -77,6 +77,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -122,7 +124,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:

google-cloud/container-linux/kubernetes/controllers.tf

@@ -23,6 +23,7 @@ locals {
   # TCP proxy load balancers require a fixed number of zonal backends. Spread
   # controllers over up to 3 zones, since all GCP regions have at least 3.
   zones = "${slice(data.google_compute_zones.all.names, 0, 3)}"
+
   controllers_ipv4_public = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.assigned_nat_ip}"]
 }
 

google-cloud/container-linux/kubernetes/network.tf

@@ -121,7 +121,7 @@ resource "google_compute_firewall" "internal-node-exporter" {
   target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }
 
-# kubelet API to allow kubectl exec and log
+# kubelet API to allow apiserver exec and log or metrics scraping
 resource "google_compute_firewall" "internal-kubelet" {
   name    = "${var.cluster_name}-internal-kubelet"
   network = "${google_compute_network.network.name}"
@@ -131,10 +131,24 @@ resource "google_compute_firewall" "internal-kubelet" {
     ports    = [10250]
   }
 
-  source_tags = ["${var.cluster_name}-controller"]
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
   target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }
 
+# Allow Prometheus to scrape ingress-controller
+resource "google_compute_firewall" "ingress-health" {
+  name    = "${var.cluster_name}-ingress-health"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "tcp"
+    ports    = [10254]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-worker"]
+}
+
 resource "google_compute_firewall" "internal-kubelet-readonly" {
   name    = "${var.cluster_name}-internal-kubelet-readonly"
   network = "${google_compute_network.network.name}"

google-cloud/container-linux/kubernetes/require.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "google" {

google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl

@@ -50,6 +50,8 @@ systemd:
         ExecStart=/usr/lib/coreos/kubelet-wrapper \
           --allow-privileged \
           --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
           --client-ca-file=/etc/kubernetes/ca.crt \
           --cluster_dns=${k8s_dns_service_ip} \
           --cluster_domain=${cluster_domain_suffix} \
@@ -92,7 +94,7 @@ storage:
       contents:
         inline: |
           KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.10.2
+          KUBELET_IMAGE_TAG=v1.10.3
     - path: /etc/sysctl.d/max-user-watches.conf
       filesystem: root
       contents:
@@ -110,7 +112,7 @@ storage:
             --volume config,kind=host,source=/etc/kubernetes \
             --mount volume=config,target=/etc/kubernetes \
             --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.10.2 \
+            docker://k8s.gcr.io/hyperkube:v1.10.3 \
             --net=host \
             --dns=host \
             --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)

google-cloud/fedora-atomic/kubernetes/README.md

@@ -18,5 +18,5 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster
 
 ## Docs
 
-Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/google-cloud/).
+Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/cl/google-cloud/).
 

google-cloud/fedora-atomic/kubernetes/apiserver.tf

@@ -14,43 +14,45 @@ resource "google_dns_record_set" "apiserver" {
 
 # Static IPv4 address for the TCP Proxy Load Balancer
 resource "google_compute_global_address" "apiserver-ipv4" {
-  name = "${var.cluster_name}-apiserver-ip"
+  name       = "${var.cluster_name}-apiserver-ip"
   ip_version = "IPV4"
 }
 
 # Forward IPv4 TCP traffic to the TCP proxy load balancer
 resource "google_compute_global_forwarding_rule" "apiserver" {
-  name = "${var.cluster_name}-apiserver"
-  ip_address = "${google_compute_global_address.apiserver-ipv4.address}"
+  name        = "${var.cluster_name}-apiserver"
+  ip_address  = "${google_compute_global_address.apiserver-ipv4.address}"
   ip_protocol = "TCP"
-  port_range = "443"
-  target = "${google_compute_target_tcp_proxy.apiserver.self_link}"
+  port_range  = "443"
+  target      = "${google_compute_target_tcp_proxy.apiserver.self_link}"
 }
 
 # TCP Proxy Load Balancer for apiservers
 resource "google_compute_target_tcp_proxy" "apiserver" {
-  name = "${var.cluster_name}-apiserver"
-  description = "Distribute TCP load across ${var.cluster_name} controllers"
+  name            = "${var.cluster_name}-apiserver"
+  description     = "Distribute TCP load across ${var.cluster_name} controllers"
   backend_service = "${google_compute_backend_service.apiserver.self_link}"
 }
 
 # Backend service backed by unmanaged instance groups
 resource "google_compute_backend_service" "apiserver" {
-  name = "${var.cluster_name}-apiserver"
+  name        = "${var.cluster_name}-apiserver"
   description = "${var.cluster_name} apiserver service"
 
-  protocol = "TCP"
-  port_name = "apiserver"
+  protocol         = "TCP"
+  port_name        = "apiserver"
   session_affinity = "NONE"
-  timeout_sec = "60"
+  timeout_sec      = "60"
 
   # controller(s) spread across zonal instance groups
   backend {
     group = "${google_compute_instance_group.controllers.0.self_link}"
   }
+
   backend {
     group = "${google_compute_instance_group.controllers.1.self_link}"
   }
+
   backend {
     group = "${google_compute_instance_group.controllers.2.self_link}"
   }
@@ -74,22 +76,22 @@ resource "google_compute_instance_group" "controllers" {
   instances = [
     "${matchkeys(google_compute_instance.controllers.*.self_link,
       google_compute_instance.controllers.*.zone,
-      list(element(local.zones, count.index)))}"
+      list(element(local.zones, count.index)))}",
   ]
 }
 
 # TCP health check for apiserver
 resource "google_compute_health_check" "apiserver" {
-  name = "${var.cluster_name}-apiserver-tcp-health"
+  name        = "${var.cluster_name}-apiserver-tcp-health"
   description = "TCP health check for kube-apiserver"
 
-  timeout_sec = 5
+  timeout_sec        = 5
   check_interval_sec = 5
 
-  healthy_threshold = 1
+  healthy_threshold   = 1
   unhealthy_threshold = 3
 
   tcp_health_check {
-    port  = "443"
+    port = "443"
   }
 }

google-cloud/fedora-atomic/kubernetes/bootkube.tf

@@ -1,6 +1,6 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=911f4115088b7511f29221f64bf8e93bfa9ee567"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=3fa3c2d73b57b2372c7c68e7db1cf82932ea1380"
 
   cluster_name          = "${var.cluster_name}"
   api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]

google-cloud/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl

@@ -54,6 +54,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -92,8 +94,8 @@ bootcmd:
 runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.4"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.5"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.12.0"
   - [systemctl, start, --no-block, etcd.service]
   - [systemctl, enable, cloud-metadata.service]

google-cloud/fedora-atomic/kubernetes/controllers.tf

@@ -23,6 +23,7 @@ locals {
   # TCP proxy load balancers require a fixed number of zonal backends. Spread
   # controllers over up to 3 zones, since all GCP regions have at least 3.
   zones = "${slice(data.google_compute_zones.all.names, 0, 3)}"
+
   controllers_ipv4_public = ["${google_compute_instance.controllers.*.network_interface.0.access_config.0.assigned_nat_ip}"]
 }
 

google-cloud/fedora-atomic/kubernetes/network.tf

@@ -121,7 +121,7 @@ resource "google_compute_firewall" "internal-node-exporter" {
   target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }
 
-# kubelet API to allow kubectl exec and log
+# kubelet API to allow apiserver exec and log or metrics scraping
 resource "google_compute_firewall" "internal-kubelet" {
   name    = "${var.cluster_name}-internal-kubelet"
   network = "${google_compute_network.network.name}"
@@ -131,10 +131,24 @@ resource "google_compute_firewall" "internal-kubelet" {
     ports    = [10250]
   }
 
-  source_tags = ["${var.cluster_name}-controller"]
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
   target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }
 
+# Allow Prometheus to scrape ingress-controller
+resource "google_compute_firewall" "ingress-health" {
+  name    = "${var.cluster_name}-ingress-health"
+  network = "${google_compute_network.network.name}"
+
+  allow {
+    protocol = "tcp"
+    ports    = [10254]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-worker"]
+}
+
 resource "google_compute_firewall" "internal-kubelet-readonly" {
   name    = "${var.cluster_name}-internal-kubelet-readonly"
   network = "${google_compute_network.network.name}"

google-cloud/fedora-atomic/kubernetes/require.tf

@@ -1,7 +1,7 @@
 # Terraform version and plugin versions
 
 terraform {
-  required_version = ">= 0.10.4"
+  required_version = ">= 0.11.0"
 }
 
 provider "google" {

google-cloud/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl

@@ -33,6 +33,8 @@ write_files:
     content: |
       ARGS="--allow-privileged \
         --anonymous-auth=false \
+        --authentication-token-webhook \
+        --authorization-mode=Webhook \
         --client-ca-file=/etc/kubernetes/ca.crt \
         --cluster_dns=${k8s_dns_service_ip} \
         --cluster_domain=${cluster_domain_suffix} \
@@ -69,7 +71,7 @@ runcmd:
   - [systemctl, daemon-reload]
   - [systemctl, restart, NetworkManager]
   - [systemctl, enable, cloud-metadata.service]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.2"
+  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.10.3"
   - [systemctl, start, --no-block, kubelet.service]
 users:
   - default