Update recommended Terraform versions and providers

* Sync the documented Terraform versions and provider plugin versions to those that are actively used/tested by the author
Promote Fedora CoreOS AWS/bare-metal to beta
2025-08-03 04:21:34 +02:00 · 2020-02-13 14:39:48 -08:00 · 2020-02-13 14:25:22 -08:00 · 2020-02-11 23:00:38 -08:00 · 2020-02-11 20:22:14 -08:00 · 2020-02-11 20:11:19 -08:00
72 changed files with 1974 additions and 180 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,31 @@ Notable changes between versions.

 ## Latest

+* Kubernetes [v1.17.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#v1173)
+* Update Calico from v3.11.2 to v3.12.0
+* Allow Fedora CoreOS clusters to pass CNCF conformance suite
+  * Set Docker log driver to `json-file` as a workaround
+
+#### AWS
+
+* Promote Fedora CoreOS to beta
+
+#### Bare-Metal
+
+* Promote Fedora CoreOS to beta
+* Add Fedora CoreOS kernel arguments initrd and console ([#640](https://github.com/poseidon/typhoon/pull/640))
+
+#### Google Cloud
+
+* Add initial Terraform module for Fedora CoreOS ([#632](https://github.com/poseidon/typhoon/pull/632))
+* Add initial support for Flatcar Container Linux ([#639](https://github.com/poseidon/typhoon/pull/639))
+
+#### Addons
+
+* Update nginx-ingress from v0.27.1 to v0.28.0
+* Update kube-state-metrics from v1.9.3 to v1.9.4
+* Update Grafana from v6.5.3 to v6.6.1
+
 ## v1.17.2

 * Kubernetes [v1.17.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.17.md#v1172)
--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@ -23,18 +23,27 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux / Flatcar Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
-| Bare-Metal    | Container Linux / Flatcar Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |

-Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/) in alpha.
+Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | alpha |
-| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | alpha |
+| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | beta |
+| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
+| Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | alpha |
+
+Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org/releases/).
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
+| Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | alpha |

 ## Documentation

@ -48,7 +57,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.3"

  # Google Cloud
  cluster_name  = "yavin"
@ -87,9 +96,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.3
 ```

 List the pods.
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -23,7 +23,7 @@ spec:
    spec:
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:6.5.3
+          image: docker.io/grafana/grafana:6.6.1
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -24,7 +24,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.9.3
+        image: quay.io/coreos/kube-state-metrics:v1.9.4
        ports:
          - name: metrics
            containerPort: 8080
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/container-linux/kubernetes/cl/controller.yaml
+++ b/aws/container-linux/kubernetes/cl/controller.yaml
@ -89,7 +89,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -134,7 +134,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            --exec=/apply
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml
@ -64,7 +64,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -128,7 +128,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            -- \
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -79,7 +79,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.3 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -123,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            k8s.gcr.io/hyperkube:v1.17.2
+            k8s.gcr.io/hyperkube:v1.17.3
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -171,10 +171,6 @@ storage:
             echo "Retry applying manifests"
             sleep 5
          done
-    - path: /etc/sysctl.d/reverse-path-filter.conf
-      contents:
-        inline: |
-          net.ipv4.conf.all.rp_filter=1
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -186,6 +182,19 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/sysconfig/docker
+      mode: 0644
+      overwrite: true
+      contents:
+        inline: |
+          # Modify these options if you want to change the way the docker daemon runs
+          OPTIONS="--selinux-enabled \
+            --log-driver=json-file \
+            --live-restore \
+            --default-ulimit nofile=1024:1024 \
+            --init-path /usr/libexec/docker/docker-init \
+            --userland-proxy-path /usr/libexec/docker/docker-proxy \
+          "
    - path: /etc/etcd/etcd.env
      mode: 0644
      contents:
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -49,7 +49,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.3 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -87,10 +87,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/sysctl.d/reverse-path-filter.conf
-      contents:
-        inline: |
-          net.ipv4.conf.all.rp_filter=1
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -102,6 +98,19 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/sysconfig/docker
+      mode: 0644
+      overwrite: true
+      contents:
+        inline: |
+          # Modify these options if you want to change the way the docker daemon runs
+          OPTIONS="--selinux-enabled \
+            --log-driver=json-file \
+            --live-restore \
+            --default-ulimit nofile=1024:1024 \
+            --init-path /usr/libexec/docker/docker-init \
+            --userland-proxy-path /usr/libexec/docker/docker-proxy \
+          "
 passwd:
  users:
    - name: core
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/azure/container-linux/kubernetes/bootstrap.tf
+++ b/azure/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/container-linux/kubernetes/cl/controller.yaml
+++ b/azure/container-linux/kubernetes/cl/controller.yaml
@ -88,7 +88,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            --exec=/apply
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml
@ -63,7 +63,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -126,7 +126,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            -- \
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/container-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml
@ -101,7 +101,7 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -147,7 +147,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            --exec=/apply
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml
@ -76,7 +76,7 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -80,7 +80,7 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.3 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -134,7 +134,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            k8s.gcr.io/hyperkube:v1.17.2
+            k8s.gcr.io/hyperkube:v1.17.3
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -182,10 +182,6 @@ storage:
             echo "Retry applying manifests"
             sleep 5
          done
-    - path: /etc/sysctl.d/reverse-path-filter.conf
-      contents:
-        inline: |
-          net.ipv4.conf.all.rp_filter=1
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -197,6 +193,19 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/sysconfig/docker
+      mode: 0644
+      overwrite: true
+      contents:
+        inline: |
+          # Modify these options if you want to change the way the docker daemon runs
+          OPTIONS="--selinux-enabled \
+            --log-driver=json-file \
+            --live-restore \
+            --default-ulimit nofile=1024:1024 \
+            --init-path /usr/libexec/docker/docker-init \
+            --userland-proxy-path /usr/libexec/docker/docker-proxy \
+          "
    - path: /etc/etcd/etcd.env
      mode: 0644
      contents:
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -50,7 +50,7 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.3 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -95,10 +95,6 @@ storage:
      contents:
        inline:
          ${domain_name}
-    - path: /etc/sysctl.d/reverse-path-filter.conf
-      contents:
-        inline: |
-          net.ipv4.conf.all.rp_filter=1
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -110,6 +106,19 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/sysconfig/docker
+      mode: 0644
+      overwrite: true
+      contents:
+        inline: |
+          # Modify these options if you want to change the way the docker daemon runs
+          OPTIONS="--selinux-enabled \
+            --log-driver=json-file \
+            --live-restore \
+            --default-ulimit nofile=1024:1024 \
+            --init-path /usr/libexec/docker/docker-init \
+            --userland-proxy-path /usr/libexec/docker/docker-proxy \
+          "
 passwd:
  users:
    - name: core
--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -4,10 +4,12 @@ locals {
  remote_args = [
    "ip=dhcp",
    "rd.neednet=1",
-    "coreos.inst=yes",
+    "initrd=fedora-coreos-${var.os_version}-live-initramfs.x86_64.img",
    "coreos.inst.image_url=https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-metal.x86_64.raw.xz",
    "coreos.inst.ignition_url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
-    "coreos.inst.install_dev=${var.install_disk}"
+    "coreos.inst.install_dev=${var.install_disk}",
+    "console=tty0",
+    "console=ttyS0",
  ]

  cached_kernel = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-kernel-x86_64"
@ -15,10 +17,12 @@ locals {
  cached_args = [
    "ip=dhcp",
    "rd.neednet=1",
-    "coreos.inst=yes",
+    "initrd=fedora-coreos-${var.os_version}-live-initramfs.x86_64.img",
    "coreos.inst.image_url=${var.matchbox_http_endpoint}/assets/fedora-coreos/fedora-coreos-${var.os_version}-metal.x86_64.raw.xz",
    "coreos.inst.ignition_url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
-    "coreos.inst.install_dev=${var.install_disk}"
+    "coreos.inst.install_dev=${var.install_disk}",
+    "console=tty0",
+    "console=ttyS0",
  ]

  kernel = var.cached_install ? local.cached_kernel : local.remote_kernel
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/digital-ocean/container-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml
@ -99,7 +99,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -144,7 +144,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            --exec=/apply
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml
@ -74,7 +74,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -132,7 +132,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            -- \
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -79,7 +79,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define

 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.17.3"
  
  # Azure
  region                  = module.ramius.region
@ -145,7 +145,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.17.3"

  # Google Cloud
  region       = "europe-west2"
@ -176,11 +176,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.17.2
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.17.2
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.17.2
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.17.2
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.17.2
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.17.3
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.17.3
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.17.3
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.17.3
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.17.3
 ```

 ### Variables
--- a/docs/announce.md
+++ b/docs/announce.md
@ -1,6 +1,6 @@
 # Announce <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo-small.png">

-## Jan 23rd, 2020
+## Jan 23, 2020

 Typhoon for Fedora CoreOS promoted to alpha!

--- a/docs/cl/aws.md
+++ b/docs/cl/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.17.2 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.3 cluster on AWS with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -49,7 +49,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf

 ```tf
 provider "aws" {
-  version                 = "2.41.0"
+  version                 = "2.48.0"
  region                  = "eu-central-1"
  shared_credentials_file = "/home/user/.config/aws/credentials"
 }
@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.17.3"

  # AWS
  cluster_name = "tempest"
@ -143,27 +143,27 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.17.2
-ip-10-0-26-65  Ready   <none>  10m  v1.17.2
-ip-10-0-41-21  Ready   <none>  10m  v1.17.2
+ip-10-0-3-155  Ready   <none>  10m  v1.17.3
+ip-10-0-26-65  Ready   <none>  10m  v1.17.3
+ip-10-0-41-21  Ready   <none>  10m  v1.17.3
 ```

 List the pods.

 ```
 $ kubectl get pods --all-namespaces
-NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE              
-kube-system   calico-node-1m5bf                         2/2    Running   0         34m              
-kube-system   calico-node-7jmr1                         2/2    Running   0         34m              
-kube-system   calico-node-bknc8                         2/2    Running   0         34m              
-kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m              
+NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
+kube-system   calico-node-1m5bf                         2/2    Running   0         34m
+kube-system   calico-node-7jmr1                         2/2    Running   0         34m
+kube-system   calico-node-bknc8                         2/2    Running   0         34m
+kube-system   coredns-1187388186-wx1lg                  1/1    Running   0         34m
 kube-system   coredns-1187388186-qjnvp                  1/1    Running   0         34m
-kube-system   kube-apiserver-ip-10-0-3-155              1/1    Running   0         34m              
-kube-system   kube-controller-manager-ip-10-0-3-155     1/1    Running   0         34m              
-kube-system   kube-proxy-14wxv                          1/1    Running   0         34m              
-kube-system   kube-proxy-9vxh2                          1/1    Running   0         34m              
-kube-system   kube-proxy-sbbsh                          1/1    Running   0         34m              
-kube-system   kube-scheduler-ip-10-0-3-155              1/1    Running   1         34m              
+kube-system   kube-apiserver-ip-10-0-3-155              1/1    Running   0         34m
+kube-system   kube-controller-manager-ip-10-0-3-155     1/1    Running   0         34m
+kube-system   kube-proxy-14wxv                          1/1    Running   0         34m
+kube-system   kube-proxy-9vxh2                          1/1    Running   0         34m
+kube-system   kube-proxy-sbbsh                          1/1    Running   0         34m
+kube-system   kube-scheduler-ip-10-0-3-155              1/1    Running   1         34m
 ```

 ## Going Further
--- a/docs/cl/azure.md
+++ b/docs/cl/azure.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.

-In this tutorial, we'll create a Kubernetes v1.17.2 cluster on Azure with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.3 cluster on Azure with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -21,7 +21,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -50,7 +50,7 @@ Configure the Azure provider in a `providers.tf` file.

 ```tf
 provider "azurerm" {
-  version = "1.38.0"
+  version = "1.43.0"
 }

 provider "ct" {
@ -66,7 +66,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.17.3"

  # Azure
  cluster_name   = "ramius"
@ -140,9 +140,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.17.2
-ramius-worker-000001  Ready   <none>  25m  v1.17.2
-ramius-worker-000002  Ready   <none>  24m  v1.17.2
+ramius-controller-0   Ready   <none>  24m  v1.17.3
+ramius-worker-000001  Ready   <none>  25m  v1.17.3
+ramius-worker-000002  Ready   <none>  24m  v1.17.3
 ```

 List the pods.
@ -152,9 +152,9 @@ $ kubectl get pods --all-namespaces
 NAMESPACE     NAME                                        READY  STATUS    RESTARTS  AGE
 kube-system   coredns-7c6fbb4f4b-b6qzx                    1/1    Running   0         26m
 kube-system   coredns-7c6fbb4f4b-j2k3d                    1/1    Running   0         26m
-kube-system   calico-node-1m5bf                           2/2    Running   0         26m              
-kube-system   calico-node-7jmr1                           2/2    Running   0         26m              
-kube-system   calico-node-bknc8                           2/2    Running   0         26m              
+kube-system   calico-node-1m5bf                           2/2    Running   0         26m
+kube-system   calico-node-7jmr1                           2/2    Running   0         26m
+kube-system   calico-node-bknc8                           2/2    Running   0         26m
 kube-system   kube-apiserver-ramius-controller-0          1/1    Running   0         26m
 kube-system   kube-controller-manager-ramius-controller-0 1/1    Running   0         26m
 kube-system   kube-proxy-j4vpq                            1/1    Running   0         26m
--- a/docs/cl/bare-metal.md
+++ b/docs/cl/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.17.2 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.17.3 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -27,7 +27,7 @@ Configure each machine to boot from the disk through IPMI or the BIOS menu.
 ```
 ipmitool -H node1 -U USER -P PASS chassis bootdev disk options=persistent
 ```
- 
+
 During provisioning, you'll explicitly set the boot device to `pxe` for the next boot only. Machines will install (overwrite) the operating system to disk on PXE boot and reboot into the disk install.

 !!! tip ""
@ -111,7 +111,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-matchbox](https://github.com/poseidon/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -160,8 +160,8 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.2"
-  
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.3"
+
  # bare-metal
  cluster_name            = "mercury"
  matchbox_http_endpoint  = "http://matchbox.example.com"
@ -299,9 +299,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.17.2
-node2.example.com   Ready   <none>  10m  v1.17.2
-node3.example.com   Ready   <none>  10m  v1.17.2
+node1.example.com   Ready   <none>  10m  v1.17.3
+node2.example.com   Ready   <none>  10m  v1.17.3
+node3.example.com   Ready   <none>  10m  v1.17.3
 ```

 List the pods.
@ -355,7 +355,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | cached_install | PXE boot and install from the Matchbox `/assets` cache. Admin MUST have downloaded Container Linux or Flatcar images into the cache | false | true |
 | install_disk | Disk device where Container Linux should be installed | "/dev/sda" | "/dev/sdb" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
-| network_mtu | CNI interface MTU (calico-only) | 1480 | - | 
+| network_mtu | CNI interface MTU (calico-only) | 1480 | - |
 | clc_snippets | Map from machine names to lists of Container Linux Config snippets | {} | [example](/advanced/customization/#usage) |
 | network_ip_autodetection_method | Method to detect host IPv4 address (calico-only) | "first-found" | "can-reach=10.0.0.1" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
--- a/docs/cl/digital-ocean.md
+++ b/docs/cl/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.17.2 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.3 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -50,7 +50,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.

 ```tf
 provider "digitalocean" {
-  version = "1.11.0"
+  version = "1.14.0"
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
 }

@ -65,7 +65,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.17.3"

  # Digital Ocean
  cluster_name = "nemo"
@ -74,7 +74,7 @@ module "nemo" {

  # configuration
  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
-  
+
  # optional
  worker_count = 2
 }
@ -138,9 +138,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.17.2
-10.132.115.81      Ready   <none>  10m  v1.17.2
-10.132.124.107     Ready   <none>  10m  v1.17.2
+10.132.110.130     Ready   <none>  10m  v1.17.3
+10.132.115.81      Ready   <none>  10m  v1.17.3
+10.132.124.107     Ready   <none>  10m  v1.17.3
 ```

 List the pods.
@ -149,9 +149,9 @@ List the pods.
 NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
 kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
 kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
-kube-system   calico-node-1m5bf                          2/2       Running   0          11m              
-kube-system   calico-node-7jmr1                          2/2       Running   0          11m              
-kube-system   calico-node-bknc8                          2/2       Running   0          11m              
+kube-system   calico-node-1m5bf                          2/2       Running   0          11m
+kube-system   calico-node-7jmr1                          2/2       Running   0          11m
+kube-system   calico-node-bknc8                          2/2       Running   0          11m
 kube-system   kube-apiserver-ip-10.132.115.81            1/1       Running   0          11m
 kube-system   kube-controller-manager-ip-10.132.115.81   1/1       Running   0          11m
 kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m
--- a/docs/cl/google-cloud.md
+++ b/docs/cl/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.17.2 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.3 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -18,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -49,7 +49,7 @@ Configure the Google Cloud provider to use your service account key, project-id,

 ```tf
 provider "google" {
-  version     = "3.4.0"
+  version     = "3.7.0"
  project     = "project-id"
  region      = "us-central1"
  credentials = file("~/.config/google-cloud/terraform.json")
@ -71,7 +71,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.3"

  # Google Cloud
  cluster_name  = "yavin"
@ -81,7 +81,7 @@ module "yavin" {

  # configuration
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-  
+
  # optional
  worker_count = 2
 }
@ -89,6 +89,31 @@ module "yavin" {

 Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.

+### Flatcar Linux Images
+
+!!! success
+    Skip this section when using CoreOS Container Linux (default). CoreOS Container Linux publishes official images to Google Cloud.
+
+!!! danger
+    Typhoon for Flatcar Linux on Google Cloud is alpha.
+
+Flatcar Linux publishes images for Google Cloud, but does not yet upload them. Google Cloud allows [custom boot images](https://cloud.google.com/compute/docs/images/import-existing-image) to be uploaded to a bucket and imported into your project.
+
+[Download](https://www.flatcar-linux.org/releases/) the Flatcar Linux GCE gzipped tarball and upload it to a Google Cloud storage bucket.
+
+```
+gsutil list
+gsutil cp flatcar_production_gce.tar.gz gs://BUCKET
+```
+
+Create a Compute Engine image from the file.
+
+```
+gcloud compute images create flatcar-linux-2303-4-0 --source-uri gs://BUCKET_NAME/flatcar_production_gce.tar.gz
+```
+
+Set the [os_image](#variables) to the image name (e.g. `flatcar-linux-2303-4-0`)
+
 ## ssh-agent

 Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
@ -145,9 +170,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.3
 ```

 List the pods.
@ -217,7 +242,7 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
-| os_image | Container Linux image for compute instances | "coreos-stable" | "coreos-stable-1632-3-0-v20180215" |
+| os_image | Container Linux image for compute instances | "coreos-stable" | "flatcar-linux-2303-4-0" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,9 +1,6 @@
 # AWS

-!!! danger
-    Typhoon for Fedora CoreOS is an alpha. Please report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues) and Typhoon issues to Typhoon.
-
-In this tutorial, we'll create a Kubernetes v1.17.2 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.17.3 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -21,7 +18,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -52,7 +49,7 @@ Configure the AWS provider to use your access key credentials in a `providers.tf

 ```tf
 provider "aws" {
-  version                 = "2.41.0"
+  version                 = "2.48.0"
  region                  = "eu-central-1"
  shared_credentials_file = "/home/user/.config/aws/credentials"
 }
@ -73,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.17.3"

  # AWS
  cluster_name = "tempest"
@ -146,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.17.2
-ip-10-0-26-65  Ready   <none>   10m  v1.17.2
-ip-10-0-41-21  Ready   <none>   10m  v1.17.2
+ip-10-0-3-155  Ready   <none>   10m  v1.17.3
+ip-10-0-26-65  Ready   <none>   10m  v1.17.3
+ip-10-0-41-21  Ready   <none>   10m  v1.17.3
 ```

 List the pods.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,9 +1,6 @@
 # Bare-Metal

-!!! danger
-    Typhoon for Fedora CoreOS is an alpha. Please report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues) and Typhoon issues to Typhoon.
-
-In this tutorial, we'll network boot and provision a Kubernetes v1.17.2 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.17.3 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -114,7 +111,7 @@ Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your sy

 ```sh
 $ terraform version
-Terraform v0.12.16
+Terraform v0.12.20
 ```

 Add the [terraform-provider-matchbox](https://github.com/poseidon/terraform-provider-matchbox) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
@ -163,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.17.3"

  # bare-metal
  cluster_name            = "mercury"
@ -292,9 +289,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.17.2
-node2.example.com   Ready   <none>  10m  v1.17.2
-node3.example.com   Ready   <none>  10m  v1.17.2
+node1.example.com   Ready   <none>  10m  v1.17.3
+node2.example.com   Ready   <none>  10m  v1.17.3
+node3.example.com   Ready   <none>  10m  v1.17.3
 ```

 List the pods.
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -0,0 +1,255 @@
+# Google Cloud
+
+!!! danger
+    Typhoon for Fedora CoreOS is an alpha. Please report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues) and Typhoon issues to Typhoon.
+
+In this tutorial, we'll create a Kubernetes v1.17.3 cluster on Google Compute Engine with Fedora CoreOS.
+
+We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.
+
+Controller hosts are provisioned to run an `etcd-member` peer and a `kubelet` service. Worker hosts run a `kubelet` service. Controller nodes run `kube-apiserver`, `kube-scheduler`, `kube-controller-manager`, and `coredns`, while `kube-proxy` and `calico` (or `flannel`) run on every node. A generated `kubeconfig` provides `kubectl` access to the cluster.
+
+## Requirements
+
+* Google Cloud Account and Service Account
+* Google Cloud DNS Zone (registered Domain Name or delegated subdomain)
+* Terraform v0.12.6+ and [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) installed locally
+
+## Terraform Setup
+
+Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your system.
+
+```sh
+$ terraform version
+Terraform v0.12.20
+```
+
+Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
+
+```sh
+wget https://github.com/poseidon/terraform-provider-ct/releases/download/v0.4.0/terraform-provider-ct-v0.4.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.4.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.4.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.4.0
+```
+
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+
+```
+cd infra/clusters
+```
+
+## Provider
+
+Login to your Google Console [API Manager](https://console.cloud.google.com/apis/dashboard) and select a project, or [signup](https://cloud.google.com/free/) if you don't have an account.
+
+Select "Credentials" and create a service account key. Choose the "Compute Engine Admin" and "DNS Administrator" roles and save the JSON private key to a file that can be referenced in configs.
+
+```sh
+mv ~/Downloads/project-id-43048204.json ~/.config/google-cloud/terraform.json
+```
+
+Configure the Google Cloud provider to use your service account key, project-id, and region in a `providers.tf` file.
+
+```tf
+provider "google" {
+  version     = "3.7.0"
+  project     = "project-id"
+  region      = "us-central1"
+  credentials = file("~/.config/google-cloud/terraform.json")
+}
+
+provider "ct" {
+  version = "0.4.0"
+}
+```
+
+Additional configuration options are described in the `google` provider [docs](https://www.terraform.io/docs/providers/google/index.html).
+
+!!! tip
+    Regions are listed in [docs](https://cloud.google.com/compute/docs/regions-zones/regions-zones) or with `gcloud compute regions list`. A project may contain multiple clusters across different regions.
+
+## Fedora CoreOS Images
+
+Fedora CoreOS publishes images for Google Cloud, but does not yet upload them. Google Cloud allows [custom boot images](https://cloud.google.com/compute/docs/images/import-existing-image) to be uploaded to a bucket and imported into your project.
+
+[Download](https://getfedora.org/coreos/download/) a Fedora CoreOS GCP gzipped tarball and upload it to a Google Cloud storage bucket.
+
+```
+gsutil list
+gsutil cp fedora-coreos-31.20200113.3.1-gcp.x86_64.tar.gz gs://BUCKET
+```
+
+Create a Compute Engine image from the file.
+
+```
+gcloud compute images create fedora-coreos-31-20200113-3-1 --source-uri gs://BUCKET/fedora-coreos-31.20200113.3.1-gcp.x86_64.tar.gz
+```
+
+## Cluster
+
+Define a Kubernetes cluster using the module `google-cloud/fedora-coreos/kubernetes`.
+
+```tf
+module "yavin" {
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=development-sha"
+
+  # Google Cloud
+  cluster_name  = "yavin"
+  region        = "us-central1"
+  dns_zone      = "example.com"
+  dns_zone_name = "example-zone"
+
+  # custom image name from above
+  os_image = "fedora-coreos-31-20200113-3-1"
+
+  # configuration
+  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+
+  # optional
+  worker_count = 2
+}
+```
+
+Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.
+
+## ssh-agent
+
+Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
+
+```sh
+ssh-add ~/.ssh/id_rsa
+ssh-add -L
+```
+
+## Apply
+
+Initialize the config directory if this is the first use with Terraform.
+
+```sh
+terraform init
+```
+
+Plan the resources to be created.
+
+```sh
+$ terraform plan
+Plan: 64 to add, 0 to change, 0 to destroy.
+```
+
+Apply the changes to create the cluster.
+
+```sh
+$ terraform apply
+module.yavin.null_resource.bootstrap: Still creating... (10s elapsed)
+...
+module.yavin.null_resource.bootstrap: Still creating... (5m30s elapsed)
+module.yavin.null_resource.bootstrap: Still creating... (5m40s elapsed)
+module.yavin.null_resource.bootstrap: Creation complete (ID: 5768638456220583358)
+
+Apply complete! Resources: 62 added, 0 changed, 0 destroyed.
+```
+
+In 4-8 minutes, the Kubernetes cluster will be ready.
+
+## Verify
+
+[Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your system. Obtain the generated cluster `kubeconfig` from module outputs (e.g. write to a local file).
+
+```
+resource "local_file" "kubeconfig-yavin" {
+  content  = module.yavin.kubeconfig-admin
+  filename = "/home/user/.kube/configs/yavin-config"
+}
+```
+
+List nodes in the cluster.
+
+```
+$ export KUBECONFIG=/home/user/.kube/configs/yavin-config
+$ kubectl get nodes
+NAME                                       ROLES    STATUS  AGE  VERSION
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.3
+```
+
+List the pods.
+
+```
+$ kubectl get pods --all-namespaces
+NAMESPACE     NAME                                      READY  STATUS    RESTARTS  AGE
+kube-system   calico-node-1cs8z                         2/2    Running   0         6m
+kube-system   calico-node-d1l5b                         2/2    Running   0         6m
+kube-system   calico-node-sp9ps                         2/2    Running   0         6m
+kube-system   coredns-1187388186-dkh3o                  1/1    Running   0         6m
+kube-system   coredns-1187388186-zj5dl                  1/1    Running   0         6m
+kube-system   kube-apiserver-controller-0               1/1    Running   0         6m
+kube-system   kube-controller-manager-controller-0      1/1    Running   0         6m
+kube-system   kube-proxy-117v6                          1/1    Running   0         6m
+kube-system   kube-proxy-9886n                          1/1    Running   0         6m
+kube-system   kube-proxy-njn47                          1/1    Running   0         6m
+kube-system   kube-scheduler-controller-0               1/1    Running   0         6m
+```
+
+## Going Further
+
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
+
+## Variables
+
+Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.
+
+### Required
+
+| Name | Description | Example |
+|:-----|:------------|:--------|
+| cluster_name | Unique cluster name (prepended to dns_zone) | "yavin" |
+| region | Google Cloud region | "us-central1" |
+| dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
+| dns_zone_name | Google Cloud DNS zone name | "example-zone" |
+| ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |
+
+Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Fedora CoreOS [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep fedora-coreos`.
+
+#### DNS Zone
+
+Clusters create a DNS A record `${cluster_name}.${dns_zone}` to resolve a TCP proxy load balancer backed by controller instances. This FQDN is used by workers and `kubectl` to access the apiserver(s). In this example, the cluster's apiserver would be accessible at `yavin.google-cloud.example.com`.
+
+You'll need a registered domain name or delegated subdomain on Google Cloud DNS. You can set this up once and create many clusters with unique names.
+
+```tf
+resource "google_dns_managed_zone" "zone-for-clusters" {
+  dns_name    = "google-cloud.example.com."
+  name        = "example-zone"
+  description = "Production DNS zone"
+}
+```
+
+!!! tip ""
+    If you have an existing domain name with a zone file elsewhere, just delegate a subdomain that can be managed on Google Cloud (e.g. google-cloud.mydomain.com) and [update nameservers](https://cloud.google.com/dns/update-name-servers).
+
+### Optional
+
+| Name | Description | Default | Example |
+|:-----|:------------|:--------|:--------|
+| asset_dir | Absolute path to a directory where generated assets should be placed (contains secrets) | "" (disabled) | "/home/user/.secrets/clusters/yavin" |
+| controller_count | Number of controllers (i.e. masters) | 1 | 3 |
+| worker_count | Number of workers | 1 | 3 |
+| controller_type | Machine type for controllers | "n1-standard-1" | See below |
+| worker_type | Machine type for workers | "n1-standard-1" | See below |
+| os_image | Fedora CoreOS image for compute instances | "" | "fedora-coreos-31-20200113-3-1" |
+| disk_size | Size of the disk in GB | 40 | 100 |
+| worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
+| controller_snippets | Controller Fedora CoreOS Config snippets | [] | UNSUPPORTED |
+| worker_snippets | Worker Fedora CoreOS Config snippets | [] | UNSUPPORTED |
+| networking | Choice of networking provider | "calico" | "calico" or "flannel" |
+| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
+| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+| worker_node_labels | List of initial worker node labels | [] | ["worker-pool=default"] |
+
+Check the list of valid [machine types](https://cloud.google.com/compute/docs/machine-types).
+
+#### Preemption
+
+Add `worker_preemptible = "true"` to allow worker nodes to be [preempted](https://cloud.google.com/compute/docs/instances/preemptible) at random, but pay [significantly](https://cloud.google.com/compute/pricing) less. Clusters tolerate stopping instances fairly well (reschedules pods, but cannot drain) and preemption provides a nice reward for running fault-tolerant cluster systems.`
+
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
@ -23,18 +23,27 @@ Typhoon provides a Terraform Module for each supported operating system and plat

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux / Flatcar Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Container Linux / Flatcar Linux | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
+| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |

-Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/) in alpha.
+Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | alpha |
-| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | alpha |
+| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | beta |
+| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | beta |
+| Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | alpha |
+
+Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org/releases/).
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](cl/aws.md) | stable |
+| Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
+| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | alpha |

 ## Documentation

@ -47,7 +56,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.3"

  # Google Cloud
  cluster_name  = "yavin"
@ -85,9 +94,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.2
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.2
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.2
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.3
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.3
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.3
 ```

 List the pods.
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -18,7 +18,7 @@ module "yavin" {
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.2"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.3"
  ...
 }
 ```
@ -279,15 +279,15 @@ Typhoon modules have been adapted for Terraform v0.12. Provider plugins requirem

 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.17.2 - ?       | v0.12.x             |
-| v1.10.3 - v1.17.2 | v0.11.x             |
+| v1.17.3 - ?       | v0.12.x             |
+| v1.10.3 - v1.17.3 | v0.11.x             |
 | v1.9.2 - v1.10.2  | v0.10.4+ or v0.11.x |
 | v1.7.3 - v1.9.1   | v0.10.x             |
 | v1.6.4 - v1.7.2   | v0.9.x              |

 ### New users

-New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.17.2+ without issue.
+New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.17.3+ without issue.

 ### Existing users

--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.2 (upstream)
+* Kubernetes v1.17.3 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/google-cloud/container-linux/kubernetes/bootstrap.tf
+++ b/google-cloud/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/container-linux/kubernetes/cl/controller.yaml
+++ b/google-cloud/container-linux/kubernetes/cl/controller.yaml
@ -88,7 +88,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            --exec=/apply
--- a/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml
@ -63,7 +63,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          docker://k8s.gcr.io/hyperkube:v1.17.3 \
          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
@ -126,7 +126,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.2 \
+            docker://k8s.gcr.io/hyperkube:v1.17.3 \
            --net=host \
            --dns=host \
            -- \
--- a/google-cloud/fedora-coreos/kubernetes/LICENSE
+++ b/google-cloud/fedora-coreos/kubernetes/LICENSE
@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Typhoon Authors
+Copyright (c) 2020 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@ -0,0 +1,23 @@
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
+
+* Kubernetes v1.17.3 (upstream)
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the Google Cloud [tutorial](https://typhoon.psdn.io/cl/google-cloud/).
+
--- a/google-cloud/fedora-coreos/kubernetes/apiserver.tf
+++ b/google-cloud/fedora-coreos/kubernetes/apiserver.tf
@ -0,0 +1,93 @@
+# TCP Proxy load balancer DNS record
+resource "google_dns_record_set" "apiserver" {
+  # DNS Zone name where record should be created
+  managed_zone = var.dns_zone_name
+
+  # DNS record
+  name = format("%s.%s.", var.cluster_name, var.dns_zone)
+  type = "A"
+  ttl  = 300
+
+  # IPv4 address of apiserver TCP Proxy load balancer
+  rrdatas = [google_compute_global_address.apiserver-ipv4.address]
+}
+
+# Static IPv4 address for the TCP Proxy Load Balancer
+resource "google_compute_global_address" "apiserver-ipv4" {
+  name       = "${var.cluster_name}-apiserver-ip"
+  ip_version = "IPV4"
+}
+
+# Forward IPv4 TCP traffic to the TCP proxy load balancer
+resource "google_compute_global_forwarding_rule" "apiserver" {
+  name        = "${var.cluster_name}-apiserver"
+  ip_address  = google_compute_global_address.apiserver-ipv4.address
+  ip_protocol = "TCP"
+  port_range  = "443"
+  target      = google_compute_target_tcp_proxy.apiserver.self_link
+}
+
+# Global TCP Proxy Load Balancer for apiservers
+resource "google_compute_target_tcp_proxy" "apiserver" {
+  name            = "${var.cluster_name}-apiserver"
+  description     = "Distribute TCP load across ${var.cluster_name} controllers"
+  backend_service = google_compute_backend_service.apiserver.self_link
+}
+
+# Global backend service backed by unmanaged instance groups
+resource "google_compute_backend_service" "apiserver" {
+  name        = "${var.cluster_name}-apiserver"
+  description = "${var.cluster_name} apiserver service"
+
+  protocol         = "TCP"
+  port_name        = "apiserver"
+  session_affinity = "NONE"
+  timeout_sec      = "300"
+
+  # controller(s) spread across zonal instance groups
+  dynamic "backend" {
+    for_each = google_compute_instance_group.controllers
+    content {
+      group = backend.value.self_link
+    }
+  }
+
+  health_checks = [google_compute_health_check.apiserver.self_link]
+}
+
+# Instance group of heterogeneous (unmanged) controller instances
+resource "google_compute_instance_group" "controllers" {
+  count = min(var.controller_count, length(local.zones))
+
+  name = format("%s-controllers-%s", var.cluster_name, element(local.zones, count.index))
+  zone = element(local.zones, count.index)
+
+  named_port {
+    name = "apiserver"
+    port = "6443"
+  }
+
+  # add instances in the zone into the instance group
+  instances = matchkeys(
+    google_compute_instance.controllers.*.self_link,
+    google_compute_instance.controllers.*.zone,
+    [element(local.zones, count.index)],
+  )
+}
+
+# TCP health check for apiserver
+resource "google_compute_health_check" "apiserver" {
+  name        = "${var.cluster_name}-apiserver-tcp-health"
+  description = "TCP health check for kube-apiserver"
+
+  timeout_sec        = 5
+  check_interval_sec = 5
+
+  healthy_threshold   = 1
+  unhealthy_threshold = 3
+
+  tcp_health_check {
+    port = "6443"
+  }
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
+++ b/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
@ -0,0 +1,22 @@
+# Kubernetes assets (kubeconfig, manifests)
+module "bootstrap" {
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=796194583426593d1a62b6f1bf3f7ffed8fca140"
+
+  cluster_name          = var.cluster_name
+  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
+  etcd_servers          = google_dns_record_set.etcds.*.name
+  asset_dir             = var.asset_dir
+  networking            = var.networking
+  network_mtu           = 1440
+  pod_cidr              = var.pod_cidr
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  enable_reporting      = var.enable_reporting
+  enable_aggregation    = var.enable_aggregation
+
+  trusted_certs_dir = "/etc/pki/tls/certs"
+
+  // temporary
+  external_apiserver_port = 443
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/controllers.tf
+++ b/google-cloud/fedora-coreos/kubernetes/controllers.tf
@ -0,0 +1,103 @@
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "google_dns_record_set" "etcds" {
+  count = var.controller_count
+
+  # DNS Zone name where record should be created
+  managed_zone = var.dns_zone_name
+
+  # DNS record
+  name = format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)
+  type = "A"
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  rrdatas = [google_compute_instance.controllers.*.network_interface.0.network_ip[count.index]]
+}
+
+# Zones in the region
+data "google_compute_zones" "all" {
+  region = var.region
+}
+
+locals {
+  zones = data.google_compute_zones.all.names
+
+  controllers_ipv4_public = google_compute_instance.controllers.*.network_interface.0.access_config.0.nat_ip
+}
+
+# Controller instances
+resource "google_compute_instance" "controllers" {
+  count = var.controller_count
+
+  name = "${var.cluster_name}-controller-${count.index}"
+  # use a zone in the region and wrap around (e.g. controllers > zones)
+  zone         = element(local.zones, count.index)
+  machine_type = var.controller_type
+
+  metadata = {
+    user-data = data.ct_config.controller-ignitions.*.rendered[count.index]
+  }
+
+  boot_disk {
+    auto_delete = true
+
+    initialize_params {
+      image = var.os_image
+      size  = var.disk_size
+    }
+  }
+
+  network_interface {
+    network = google_compute_network.network.name
+
+    # Ephemeral external IP
+    access_config {
+    }
+  }
+
+  can_ip_forward = true
+  tags           = ["${var.cluster_name}-controller"]
+
+  lifecycle {
+    ignore_changes = [metadata]
+  }
+}
+
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count    = var.controller_count
+  content  = data.template_file.controller-configs.*.rendered[count.index]
+  strict   = true
+  snippets = var.controller_snippets
+}
+
+# Controller Fedora CoreOS configs
+data "template_file" "controller-configs" {
+  count = var.controller_count
+
+  template = file("${path.module}/fcc/controller.yaml")
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+  }
+}
+
+data "template_file" "etcds" {
+  count    = var.controller_count
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+  vars = {
+    index        = count.index
+    cluster_name = var.cluster_name
+    dns_zone     = var.dns_zone
+  }
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
@ -0,0 +1,225 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/coreos/etcd
+        Wants=network-online.target network.target
+        After=network-online.target
+        [Service]
+        # https://github.com/opencontainers/runc/pull/1807
+        # Type=notify
+        # NotifyAccess=exec
+        Type=exec
+        Restart=on-failure
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        ExecStartPre=/bin/mkdir -p /var/lib/etcd
+        ExecStartPre=-/usr/bin/podman rm etcd
+        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
+        ExecStart=/usr/bin/podman run --name etcd \
+          --env-file /etc/etcd/etcd.env \
+          --network host \
+          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
+          quay.io/coreos/etcd:v3.4.3
+        ExecStop=/usr/bin/podman stop etcd
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube (System Container)
+        Wants=rpc-statd.service
+        [Service]
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/podman rm kubelet
+        ExecStart=/usr/bin/podman run --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /usr/lib/os-release:/etc/os-release:ro \
+          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /lib/modules:/lib/modules:ro \
+          --volume /run:/run \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /var/lib/calico:/var/lib/calico \
+          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+          --volume /var/log:/var/log \
+          --volume /var/run/lock:/var/run/lock:z \
+          --volume /opt/cni/bin:/opt/cni/bin:z \
+          k8s.gcr.io/hyperkube:v1.17.3 kubelet \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --cgroup-driver=systemd \
+          --cgroups-per-qos=true \
+          --enforce-node-allocatable=pods \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --healthz-port=0 \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/master \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/podman stop kubelet
+        Delegate=yes
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/bash -c 'set -x && [ -n "$(ls /opt/bootstrap/assets/manifests-*/* 2>/dev/null)" ] && mv /opt/bootstrap/assets/manifests-*/* /opt/bootstrap/assets/manifests && rm -rf /opt/bootstrap/assets/manifests-*'
+        ExecStart=/usr/bin/podman run --name bootstrap \
+            --network host \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /opt/bootstrap/assets:/assets:ro,Z \
+            --volume /opt/bootstrap/apply:/apply:ro,Z \
+            --entrypoint=/apply \
+            k8s.gcr.io/hyperkube:v1.17.3
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        ExecStartPost=-/usr/bin/podman stop bootstrap
+storage:
+  directories:
+    - path: /etc/kubernetes
+    - path: /opt/bootstrap
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /opt/bootstrap/layout
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          sudo mkdir -p /etc/kubernetes/manifests
+          sudo mv static-manifests/* /etc/kubernetes/manifests/
+          sudo mkdir -p /opt/bootstrap/assets
+          sudo mv manifests /opt/bootstrap/assets/manifests
+          sudo mv manifests-networking /opt/bootstrap/assets/manifests-networking
+          rm -rf assets auth static-manifests tls
+    - path: /opt/bootstrap/apply
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/sysctl.d/max-user-watches.conf
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/systemd/system.conf.d/accounting.conf
+      contents:
+        inline: |
+          [Manager]
+          DefaultCPUAccounting=yes
+          DefaultMemoryAccounting=yes
+          DefaultBlockIOAccounting=yes
+    - path: /etc/sysconfig/docker
+      mode: 0644
+      overwrite: true
+      contents:
+        inline: |
+          # Modify these options if you want to change the way the docker daemon runs
+          OPTIONS="--selinux-enabled \
+            --log-driver=json-file \
+            --live-restore \
+            --default-ulimit nofile=1024:1024 \
+            --init-path /usr/libexec/docker/docker-init \
+            --userland-proxy-path /usr/libexec/docker/docker-proxy \
+          "
+    - path: /etc/etcd/etcd.env
+      mode: 0644
+      contents:
+        inline: |
+          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
+          NOTIFY_SOCKET=/run/systemd/notify
+          ETCD_NAME=${etcd_name}
+          ETCD_DATA_DIR=/var/lib/etcd
+          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+          ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+          ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+          ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+          ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+          ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+          ETCD_STRICT_RECONFIG_CHECK=true
+          ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+          ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+          ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+          ETCD_CLIENT_CERT_AUTH=true
+          ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+          ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - ${ssh_authorized_key}
--- a/google-cloud/fedora-coreos/kubernetes/ingress.tf
+++ b/google-cloud/fedora-coreos/kubernetes/ingress.tf
@ -0,0 +1,123 @@
+# Static IPv4 address for Ingress Load Balancing
+resource "google_compute_global_address" "ingress-ipv4" {
+  name       = "${var.cluster_name}-ingress-ipv4"
+  ip_version = "IPV4"
+}
+
+# Static IPv6 address for Ingress Load Balancing
+resource "google_compute_global_address" "ingress-ipv6" {
+  name       = "${var.cluster_name}-ingress-ipv6"
+  ip_version = "IPV6"
+}
+
+# Forward IPv4 TCP traffic to the HTTP proxy load balancer
+# Google Cloud does not allow TCP proxies for port 80. Must use HTTP proxy.
+resource "google_compute_global_forwarding_rule" "ingress-http-ipv4" {
+  name        = "${var.cluster_name}-ingress-http-ipv4"
+  ip_address  = google_compute_global_address.ingress-ipv4.address
+  ip_protocol = "TCP"
+  port_range  = "80"
+  target      = google_compute_target_http_proxy.ingress-http.self_link
+}
+
+# Forward IPv4 TCP traffic to the TCP proxy load balancer
+resource "google_compute_global_forwarding_rule" "ingress-https-ipv4" {
+  name        = "${var.cluster_name}-ingress-https-ipv4"
+  ip_address  = google_compute_global_address.ingress-ipv4.address
+  ip_protocol = "TCP"
+  port_range  = "443"
+  target      = google_compute_target_tcp_proxy.ingress-https.self_link
+}
+
+# Forward IPv6 TCP traffic to the HTTP proxy load balancer
+# Google Cloud does not allow TCP proxies for port 80. Must use HTTP proxy.
+resource "google_compute_global_forwarding_rule" "ingress-http-ipv6" {
+  name        = "${var.cluster_name}-ingress-http-ipv6"
+  ip_address  = google_compute_global_address.ingress-ipv6.address
+  ip_protocol = "TCP"
+  port_range  = "80"
+  target      = google_compute_target_http_proxy.ingress-http.self_link
+}
+
+# Forward IPv6 TCP traffic to the TCP proxy load balancer
+resource "google_compute_global_forwarding_rule" "ingress-https-ipv6" {
+  name        = "${var.cluster_name}-ingress-https-ipv6"
+  ip_address  = google_compute_global_address.ingress-ipv6.address
+  ip_protocol = "TCP"
+  port_range  = "443"
+  target      = google_compute_target_tcp_proxy.ingress-https.self_link
+}
+
+# HTTP proxy load balancer for ingress controllers
+resource "google_compute_target_http_proxy" "ingress-http" {
+  name        = "${var.cluster_name}-ingress-http"
+  description = "Distribute HTTP load across ${var.cluster_name} workers"
+  url_map     = google_compute_url_map.ingress-http.self_link
+}
+
+# TCP proxy load balancer for ingress controllers
+resource "google_compute_target_tcp_proxy" "ingress-https" {
+  name            = "${var.cluster_name}-ingress-https"
+  description     = "Distribute HTTPS load across ${var.cluster_name} workers"
+  backend_service = google_compute_backend_service.ingress-https.self_link
+}
+
+# HTTP URL Map (required)
+resource "google_compute_url_map" "ingress-http" {
+  name = "${var.cluster_name}-ingress-http"
+
+  # Do not add host/path rules for applications here. Use Ingress resources.
+  default_service = google_compute_backend_service.ingress-http.self_link
+}
+
+# Backend service backed by managed instance group of workers
+resource "google_compute_backend_service" "ingress-http" {
+  name        = "${var.cluster_name}-ingress-http"
+  description = "${var.cluster_name} ingress service"
+
+  protocol         = "HTTP"
+  port_name        = "http"
+  session_affinity = "NONE"
+  timeout_sec      = "60"
+
+  backend {
+    group = module.workers.instance_group
+  }
+
+  health_checks = [google_compute_health_check.ingress.self_link]
+}
+
+# Backend service backed by managed instance group of workers
+resource "google_compute_backend_service" "ingress-https" {
+  name        = "${var.cluster_name}-ingress-https"
+  description = "${var.cluster_name} ingress service"
+
+  protocol         = "TCP"
+  port_name        = "https"
+  session_affinity = "NONE"
+  timeout_sec      = "60"
+
+  backend {
+    group = module.workers.instance_group
+  }
+
+  health_checks = [google_compute_health_check.ingress.self_link]
+}
+
+# Ingress HTTP Health Check
+resource "google_compute_health_check" "ingress" {
+  name        = "${var.cluster_name}-ingress-health"
+  description = "Health check for Ingress controller"
+
+  timeout_sec        = 5
+  check_interval_sec = 5
+
+  healthy_threshold   = 2
+  unhealthy_threshold = 4
+
+  http_health_check {
+    port         = 10254
+    request_path = "/healthz"
+  }
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/network.tf
+++ b/google-cloud/fedora-coreos/kubernetes/network.tf
@ -0,0 +1,193 @@
+resource "google_compute_network" "network" {
+  name                    = var.cluster_name
+  description             = "Network for the ${var.cluster_name} cluster"
+  auto_create_subnetworks = true
+
+  timeouts {
+    delete = "6m"
+  }
+}
+
+resource "google_compute_firewall" "allow-ssh" {
+  name    = "${var.cluster_name}-allow-ssh"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [22]
+  }
+
+  source_ranges = ["0.0.0.0/0"]
+  target_tags   = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+resource "google_compute_firewall" "internal-etcd" {
+  name    = "${var.cluster_name}-internal-etcd"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [2379, 2380]
+  }
+
+  source_tags = ["${var.cluster_name}-controller"]
+  target_tags = ["${var.cluster_name}-controller"]
+}
+
+# Allow Prometheus to scrape etcd metrics
+resource "google_compute_firewall" "internal-etcd-metrics" {
+  name    = "${var.cluster_name}-internal-etcd-metrics"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [2381]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller"]
+}
+
+# Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
+resource "google_compute_firewall" "internal-kube-metrics" {
+  name    = "${var.cluster_name}-internal-kube-metrics"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [10251, 10252]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller"]
+}
+
+resource "google_compute_firewall" "allow-apiserver" {
+  name    = "${var.cluster_name}-allow-apiserver"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [6443]
+  }
+
+  source_ranges = ["0.0.0.0/0"]
+  target_tags   = ["${var.cluster_name}-controller"]
+}
+
+# BGP and IPIP
+# https://docs.projectcalico.org/latest/reference/public-cloud/gce
+resource "google_compute_firewall" "internal-bgp" {
+  count = var.networking != "flannel" ? 1 : 0
+
+  name    = "${var.cluster_name}-internal-bgp"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = ["179"]
+  }
+
+  allow {
+    protocol = "ipip"
+  }
+
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# flannel VXLAN
+resource "google_compute_firewall" "internal-vxlan" {
+  count = var.networking == "flannel" ? 1 : 0
+
+  name    = "${var.cluster_name}-internal-vxlan"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "udp"
+    ports    = [4789]
+  }
+
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# Allow Prometheus to scrape node-exporter daemonset
+resource "google_compute_firewall" "internal-node-exporter" {
+  name    = "${var.cluster_name}-internal-node-exporter"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [9100]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# Allow Prometheus to scrape kube-proxy metrics
+resource "google_compute_firewall" "internal-kube-proxy" {
+  name    = "${var.cluster_name}-internal-kube-proxy"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [10249]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# Allow apiserver to access kubelets for exec, log, port-forward
+resource "google_compute_firewall" "internal-kubelet" {
+  name    = "${var.cluster_name}-internal-kubelet"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [10250]
+  }
+
+  # allow Prometheus to scrape kubelet metrics too
+  source_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
+# Workers
+
+resource "google_compute_firewall" "allow-ingress" {
+  name    = "${var.cluster_name}-allow-ingress"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [80, 443]
+  }
+
+  source_ranges = ["0.0.0.0/0"]
+  target_tags   = ["${var.cluster_name}-worker"]
+}
+
+resource "google_compute_firewall" "google-ingress-health-checks" {
+  name    = "${var.cluster_name}-ingress-health"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [10254]
+  }
+
+  # https://cloud.google.com/load-balancing/docs/health-check-concepts#method
+  source_ranges = [
+    "35.191.0.0/16",
+    "130.211.0.0/22",
+    "35.191.0.0/16",
+    "209.85.152.0/22",
+    "209.85.204.0/22",
+  ]
+
+  target_tags = ["${var.cluster_name}-worker"]
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/outputs.tf
+++ b/google-cloud/fedora-coreos/kubernetes/outputs.tf
@ -0,0 +1,44 @@
+output "kubeconfig-admin" {
+  value = module.bootstrap.kubeconfig-admin
+}
+
+# Outputs for Kubernetes Ingress
+
+output "ingress_static_ipv4" {
+  description = "Global IPv4 address for proxy load balancing to the nearest Ingress controller"
+  value       = google_compute_global_address.ingress-ipv4.address
+}
+
+output "ingress_static_ipv6" {
+  description = "Global IPv6 address for proxy load balancing to the nearest Ingress controller"
+  value       = google_compute_global_address.ingress-ipv6.address
+}
+
+# Outputs for worker pools
+
+output "network_name" {
+  value = google_compute_network.network.name
+}
+
+output "kubeconfig" {
+  value = module.bootstrap.kubeconfig-kubelet
+}
+
+# Outputs for custom firewalling
+
+output "network_self_link" {
+  value = google_compute_network.network.self_link
+}
+
+# Outputs for custom load balancing
+
+output "worker_instance_group" {
+  description = "Worker managed instance group full URL"
+  value       = module.workers.instance_group
+}
+
+output "worker_target_pool" {
+  description = "Worker target pool self link"
+  value       = module.workers.target_pool
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/ssh.tf
+++ b/google-cloud/fedora-coreos/kubernetes/ssh.tf
@ -0,0 +1,58 @@
+locals {
+  # format assets for distribution
+  assets_bundle = [
+    # header with the unpack location
+    for key, value in module.bootstrap.assets_dist :
+    format("##### %s\n%s", key, value)
+  ]
+}
+
+# Secure copy assets to controllers.
+resource "null_resource" "copy-controller-secrets" {
+  count = var.controller_count
+
+  depends_on = [
+    module.bootstrap,
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = local.controllers_ipv4_public[count.index]
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = join("\n", local.assets_bundle)
+    destination = "$HOME/assets"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo /opt/bootstrap/layout",
+    ]
+  }
+}
+
+# Connect to a controller to perform one-time cluster bootstrap.
+resource "null_resource" "bootstrap" {
+  depends_on = [
+    null_resource.copy-controller-secrets,
+    module.workers,
+    google_dns_record_set.apiserver,
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = local.controllers_ipv4_public[0]
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo systemctl start bootstrap",
+    ]
+  }
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/variables.tf
+++ b/google-cloud/fedora-coreos/kubernetes/variables.tf
@ -0,0 +1,138 @@
+variable "cluster_name" {
+  type        = string
+  description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# Google Cloud
+
+variable "region" {
+  type        = string
+  description = "Google Cloud Region (e.g. us-central1, see `gcloud compute regions list`)"
+}
+
+variable "dns_zone" {
+  type        = string
+  description = "Google Cloud DNS Zone (e.g. google-cloud.example.com)"
+}
+
+variable "dns_zone_name" {
+  type        = string
+  description = "Google Cloud DNS Zone name (e.g. example-zone)"
+}
+
+# instances
+
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "Machine type for controllers (see `gcloud compute machine-types list`)"
+  default     = "n1-standard-1"
+}
+
+variable "worker_type" {
+  type        = string
+  description = "Machine type for controllers (see `gcloud compute machine-types list`)"
+  default     = "n1-standard-1"
+}
+
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for compute instances (e.g. fedora-coreos)"
+}
+
+variable "disk_size" {
+  type        = number
+  description = "Size of the disk in GB"
+  default     = 40
+}
+
+variable "worker_preemptible" {
+  type        = bool
+  description = "If enabled, Compute Engine will terminate workers randomly within 24 hours"
+  default     = false
+}
+
+variable "controller_snippets" {
+  type        = list(string)
+  description = "Controller Fedora CoreOS Config snippets"
+  default     = []
+}
+
+variable "worker_snippets" {
+  type        = list(string)
+  description = "Worker Fedora CoreOS Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "ssh_authorized_key" {
+  type        = string
+  description = "SSH public key for user 'core'"
+}
+
+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
+variable "networking" {
+  type        = string
+  description = "Choice of networking provider (flannel or calico)"
+  default     = "calico"
+}
+
+variable "pod_cidr" {
+  type        = string
+  description = "CIDR IPv4 range to assign Kubernetes pods"
+  default     = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+  type        = string
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+  default     = "10.3.0.0/16"
+}
+
+
+variable "enable_reporting" {
+  type        = bool
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = false
+}
+
+variable "enable_aggregation" {
+  type        = bool
+  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  default     = false
+}
+
+variable "worker_node_labels" {
+  type        = list(string)
+  description = "List of initial worker node labels"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "cluster_domain_suffix" {
+  type        = string
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  default     = "cluster.local"
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/versions.tf
+++ b/google-cloud/fedora-coreos/kubernetes/versions.tf
@ -0,0 +1,11 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.6"
+  required_providers {
+    google   = ">= 2.19, < 4.0"
+    ct       = "~> 0.3"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}
--- a/google-cloud/fedora-coreos/kubernetes/workers.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers.tf
@ -0,0 +1,23 @@
+module "workers" {
+  source       = "./workers"
+  name         = var.cluster_name
+  cluster_name = var.cluster_name
+
+  # GCE
+  region       = var.region
+  network      = google_compute_network.network.name
+  worker_count = var.worker_count
+  machine_type = var.worker_type
+  os_image     = var.os_image
+  disk_size    = var.disk_size
+  preemptible  = var.worker_preemptible
+
+  # configuration
+  kubeconfig            = module.bootstrap.kubeconfig-kubelet
+  ssh_authorized_key    = var.ssh_authorized_key
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  snippets              = var.worker_snippets
+  node_labels           = var.worker_node_labels
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -0,0 +1,119 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+  units:
+    - name: docker.service
+      enabled: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube (System Container)
+        Wants=rpc-statd.service
+        [Service]
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/podman rm kubelet
+        ExecStart=/usr/bin/podman run --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /usr/lib/os-release:/etc/os-release:ro \
+          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /lib/modules:/lib/modules:ro \
+          --volume /run:/run \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /var/lib/calico:/var/lib/calico \
+          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+          --volume /var/log:/var/log \
+          --volume /var/run/lock:/var/run/lock:z \
+          --volume /opt/cni/bin:/opt/cni/bin:z \
+          k8s.gcr.io/hyperkube:v1.17.3 kubelet \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --cgroup-driver=systemd \
+          --cgroups-per-qos=true \
+          --enforce-node-allocatable=pods \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --healthz-port=0 \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/node \
+          %{ for label in split(",", node_labels) }
+          --node-labels=${label} \
+          %{ endfor ~}
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/podman stop kubelet
+        Delegate=yes
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /etc/kubernetes
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/sysctl.d/max-user-watches.conf
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/systemd/system.conf.d/accounting.conf
+      contents:
+        inline: |
+          [Manager]
+          DefaultCPUAccounting=yes
+          DefaultMemoryAccounting=yes
+          DefaultBlockIOAccounting=yes
+    - path: /etc/sysconfig/docker
+      mode: 0644
+      overwrite: true
+      contents:
+        inline: |
+          # Modify these options if you want to change the way the docker daemon runs
+          OPTIONS="--selinux-enabled \
+            --log-driver=json-file \
+            --live-restore \
+            --default-ulimit nofile=1024:1024 \
+            --init-path /usr/libexec/docker/docker-init \
+            --userland-proxy-path /usr/libexec/docker/docker-proxy \
+          "
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - ${ssh_authorized_key}
+
--- a/google-cloud/fedora-coreos/kubernetes/workers/outputs.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/outputs.tf
@ -0,0 +1,14 @@
+# Outputs for global load balancing
+
+output "instance_group" {
+  description = "Worker managed instance group full URL"
+  value       = google_compute_region_instance_group_manager.workers.instance_group
+}
+
+# Outputs for regional load balancing
+
+output "target_pool" {
+  description = "Worker target pool self link"
+  value       = google_compute_target_pool.workers.self_link
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/workers/target_pool.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/target_pool.tf
@ -0,0 +1,23 @@
+# Target pool for TCP/UDP load balancing
+resource "google_compute_target_pool" "workers" {
+  name             = "${var.name}-worker-pool"
+  region           = var.region
+  session_affinity = "NONE"
+
+  health_checks = [
+    google_compute_http_health_check.workers.name,
+  ]
+}
+
+# HTTP Health Check (for TCP/UDP load balancing)
+# Forward rules (regional) to target pools don't support different external
+# and internal ports. Health check for nodes with Ingress controllers that
+# may support proxying or otherwise satisfy the check.
+resource "google_compute_http_health_check" "workers" {
+  name        = "${var.name}-target-pool-health"
+  description = "Health check for the worker target pool"
+
+  port         = 10254
+  request_path = "/healthz"
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/workers/variables.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/variables.tf
@ -0,0 +1,106 @@
+variable "name" {
+  type        = string
+  description = "Unique name for the worker pool"
+}
+
+variable "cluster_name" {
+  type        = string
+  description = "Must be set to `cluster_name of cluster`"
+}
+
+# Google Cloud
+
+variable "region" {
+  type        = string
+  description = "Must be set to `region` of cluster"
+}
+
+variable "network" {
+  type        = string
+  description = "Must be set to `network_name` output by cluster"
+}
+
+# instances
+
+variable "worker_count" {
+  type        = number
+  description = "Number of worker compute instances the instance group should manage"
+  default     = 1
+}
+
+variable "machine_type" {
+  type        = string
+  description = "Machine type for compute instances (e.g. gcloud compute machine-types list)"
+  default     = "n1-standard-1"
+}
+
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for compute instanges (e.g. gcloud compute images list)"
+}
+
+variable "disk_size" {
+  type        = number
+  description = "Size of the disk in GB"
+  default     = 40
+}
+
+variable "preemptible" {
+  type        = bool
+  description = "If enabled, Compute Engine will terminate instances randomly within 24 hours"
+  default     = false
+}
+
+variable "snippets" {
+  type        = list(string)
+  description = "Fedora CoreOS Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "kubeconfig" {
+  type        = string
+  description = "Must be set to `kubeconfig` output by cluster"
+}
+
+variable "ssh_authorized_key" {
+  type        = string
+  description = "SSH public key for user 'core'"
+}
+
+variable "service_cidr" {
+  type        = string
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+  default     = "10.3.0.0/16"
+}
+
+variable "node_labels" {
+  type        = list(string)
+  description = "List of initial node labels"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported, temporary
+
+variable "cluster_domain_suffix" {
+  type        = string
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  default     = "cluster.local"
+}
+
+variable "accelerator_type" {
+  type        = string
+  default     = ""
+  description = "Google Compute Engine accelerator type (e.g. nvidia-tesla-k80, see gcloud compute accelerator-types list)"
+}
+
+variable "accelerator_count" {
+  type        = string
+  default     = "0"
+  description = "Number of compute engine accelerators"
+}
+
--- a/google-cloud/fedora-coreos/kubernetes/workers/versions.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/versions.tf
@ -0,0 +1,4 @@
+
+terraform {
+  required_version = ">= 0.12"
+}
--- a/google-cloud/fedora-coreos/kubernetes/workers/workers.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/workers.tf
@ -0,0 +1,91 @@
+# Managed instance group of workers
+resource "google_compute_region_instance_group_manager" "workers" {
+  name        = "${var.name}-worker-group"
+  description = "Compute instance group of ${var.name} workers"
+
+  # instance name prefix for instances in the group
+  base_instance_name = "${var.name}-worker"
+  region             = var.region
+  version {
+    name              = "default"
+    instance_template = google_compute_instance_template.worker.self_link
+  }
+
+  target_size  = var.worker_count
+  target_pools = [google_compute_target_pool.workers.self_link]
+
+  named_port {
+    name = "http"
+    port = "80"
+  }
+
+  named_port {
+    name = "https"
+    port = "443"
+  }
+}
+
+# Worker instance template
+resource "google_compute_instance_template" "worker" {
+  name_prefix  = "${var.name}-worker-"
+  description  = "Worker Instance template"
+  machine_type = var.machine_type
+
+  metadata = {
+    user-data = data.ct_config.worker-ignition.rendered
+  }
+
+  scheduling {
+    automatic_restart = var.preemptible ? false : true
+    preemptible       = var.preemptible
+  }
+
+  disk {
+    auto_delete  = true
+    boot         = true
+    source_image = var.os_image
+    disk_size_gb = var.disk_size
+  }
+
+  network_interface {
+    network = var.network
+
+    # Ephemeral external IP
+    access_config {
+    }
+  }
+
+  can_ip_forward = true
+  tags           = ["worker", "${var.cluster_name}-worker", "${var.name}-worker"]
+
+  guest_accelerator {
+    count = var.accelerator_count
+    type  = var.accelerator_type
+  }
+
+  lifecycle {
+    # To update an Instance Template, Terraform should replace the existing resource
+    create_before_destroy = true
+  }
+}
+
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = data.template_file.worker-config.rendered
+  strict       = true
+  snippets     = var.snippets
+}
+
+# Worker Fedora CoreOS config
+data "template_file" "worker-config" {
+  template = file("${path.module}/fcc/worker.yaml")
+
+  vars = {
+    kubeconfig             = indent(10, var.kubeconfig)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    node_labels            = join(",", var.node_labels)
+  }
+}
+
--- a/google-cloud/ignore/.gitkeep
+++ b/google-cloud/ignore/.gitkeep
--- a/mkdocs.yml
+++ b/mkdocs.yml
@ -1,5 +1,5 @@
 site_name: 'Typhoon'
-site_description: 'A minimal and free Kubernetes distribution' 
+site_description: 'A minimal and free Kubernetes distribution'
 site_author: 'Dalton Hubble'
 repo_name: 'poseidon/typhoon'
 repo_url: 'https://github.com/poseidon/typhoon'
@ -53,15 +53,16 @@ nav:
    - 'Bare-Metal': 'architecture/bare-metal.md'
    - 'DigitalOcean': 'architecture/digitalocean.md'
    - 'Google Cloud': 'architecture/google-cloud.md'
-  - 'Container Linux': 
+  - 'Container Linux':
    - 'AWS': 'cl/aws.md'
    - 'Azure': 'cl/azure.md'
    - 'Bare-Metal': 'cl/bare-metal.md'
    - 'Digital Ocean': 'cl/digital-ocean.md'
    - 'Google Cloud': 'cl/google-cloud.md'
-  - 'Fedora CoreOS': 
+  - 'Fedora CoreOS':
    - 'AWS': 'fedora-coreos/aws.md'
    - 'Bare-Metal': 'fedora-coreos/bare-metal.md'
+    - 'Google Cloud': 'fedora-coreos/google-cloud.md'
  - 'Topics':
    - 'Maintenance': 'topics/maintenance.md'
    - 'Hardware': 'topics/hardware.md'
--- a/requirements.txt
+++ b/requirements.txt
@ -1,4 +1,4 @@
 mkdocs==1.0.4
-mkdocs-material==4.6.0
-pygments==2.4.2
-pymdown-extensions==6.2.0
+mkdocs-material==4.6.2
+pygments==2.5.2
+pymdown-extensions==6.3.0
Author	SHA1	Message	Date
Dalton Hubble	0c53ad52e4	Update recommended Terraform versions and providers * Sync the documented Terraform versions and provider plugin versions to those that are actively used/tested by the author	2020-02-13 14:39:48 -08:00
Dalton Hubble	008817b0aa	Promote Fedora CoreOS AWS/bare-metal to beta * Remove alpha warnings from docs headers	2020-02-13 14:25:22 -08:00
Dalton Hubble	49d3b9e6b3	Set docker log driver to json-file on Fedora CoreOS * Fix the last minor issue for Fedora CoreOS clusters to pass CNCF's Kubernetes conformance tests * Kubelet supports a seldom used feature `kubectl logs --limit-bytes=N` to trim a log stream to a desired length. Kubelet handles this in the CRI driver. The Kubelet docker shim only supports the limit bytes feature when Docker is configured with the default `json-file` logging driver * CNCF conformance tests started requiring limit-bytes be supported, indirectly forcing the log driver choice until either the Kubelet or the conformance tests are fixed * Fedora CoreOS defaults Docker to use `journald` (desired). For now, as a workaround to offer conformant clusters, the log driver can be set back to `json-file`. RHEL CoreOS likely won't have noticed the non-conformance since its using crio runtime * https://github.com/kubernetes/kubernetes/issues/86367 Note: When upstream has a fix, the aim is to drop the docker config override and use the journald default	2020-02-11 23:00:38 -08:00
Dalton Hubble	1243f395d1	Update Kubernetes from v1.17.2 to v1.17.3 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#v1173	2020-02-11 20:22:14 -08:00
Dalton Hubble	846f11097f	Update Fedora CoreOS kernel arguments to align with upstream * Align bare-metal kernel arguments with upstream docs * Add missing initrd argument which can cause issues if not present. Fix #638 * Add tty0 and ttyS0 consoles (matches Container Linux) * Remove unused coreos.inst=yes Related: https://docs.fedoraproject.org/en-US/fedora-coreos/bare-metal/	2020-02-11 20:11:19 -08:00
Dalton Hubble	ba84f86dc7	Add guide for Typhoon with Flatcar Linux on Google Cloud * Add docs on manually uploading a Flatcar Linux GCE/GCP gzipped tarball image as a Compute Engine image for use with the Typhoon container-linux module * Set status of Flatcar Linux on Google Cloud to alpha	2020-02-11 19:38:40 -08:00
Dalton Hubble	b49a1d715d	Update docs generation packages * Update mkdocs-material from v4.6.0 to v4.6.2	2020-02-08 15:12:12 -08:00
Dalton Hubble	34c3d7cc39	Update Grafana from v6.6.0 to v6.6.1 * https://github.com/grafana/grafana/releases/tag/v6.6.1	2020-02-08 14:50:33 -08:00
Dalton Hubble	ca96a1335c	Update Calico from v3.11.2 to v3.12.0 * https://docs.projectcalico.org/release-notes/#v3120 * Remove reverse packet filter override, since Calico no longer relies on the setting * https://github.com/coreos/fedora-coreos-tracker/issues/219 * https://github.com/projectcalico/felix/pull/2189	2020-02-06 00:43:33 -08:00
Dalton Hubble	e339fbd2b6	Update kube-state-metrics from v1.9.3 to v1.9.4 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.4	2020-02-04 21:33:34 -08:00
Dalton Hubble	8cc303c9ac	Add module for Fedora CoreOS on Google Cloud * Add Typhoon Fedora CoreOS on Google Cloud as alpha * Add docs on uploading the Fedora CoreOS GCP gzipped tarball to Google Cloud storage to create a boot disk image	2020-02-01 15:21:40 -08:00
Dalton Hubble	b19ba16afa	Update nginx-ingress from v0.27.1 to v0.28.0 * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.28.0	2020-01-30 18:00:23 -08:00
Dalton Hubble	d127a7345c	Update Grafana from v6.5.3 to v6.6.0 * https://github.com/grafana/grafana/releases/tag/v6.6.0	2020-01-27 20:46:32 -08:00
Dalton Hubble	02a470d2f2	Fix minor typo in announcement date	2020-01-23 08:57:01 -08:00