Promote Fedora CoreOS from preview to alpha in docs

* Add an announcement to the website as well
Update kube-state-metrics from v1.9.2 to v1.9.3
2025-08-02 19:01:34 +02:00 · 2020-01-23 08:47:18 -08:00 · 2020-01-23 00:03:16 -08:00 · 2020-01-21 18:27:39 -08:00 · 2020-01-20 15:22:16 -08:00 · 2020-01-20 14:44:06 -08:00
88 changed files with 790 additions and 485 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,54 @@ Notable changes between versions.

 ## Latest

+## v1.17.2
+
+* Kubernetes [v1.17.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.17.md#v1172)
+
+#### AWS
+
+* Promote Fedora CoreOS from preview to alpha
+
+#### Bare-Metal
+
+* Promote Fedora CoreOS from preview to alpha
+* Update Fedora CoreOS images location
+  * Use Fedora CoreOS production [download](https://getfedora.org/coreos/download/) streams
+  * Use live PXE kernel and initramfs images
+
+#### Addons
+
+* Update nginx-ingress from v0.26.1 to [v0.27.1](https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.27.1) ([#625](https://github.com/poseidon/typhoon/pull/625))
+  * Change runAsUser from 33 to 101 for alpine-based image
+* Update kube-state-metrics from v1.9.2 to v1.9.3
+
+## v1.17.1
+
+* Kubernetes [v1.17.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.17.md#v1171)
+* Update CoreDNS from v1.6.5 to [v1.6.6](https://coredns.io/2019/12/11/coredns-1.6.6-release/) ([#602](https://github.com/poseidon/typhoon/pull/602))
+* Update Calico from v3.10.2 to v3.11.2 ([#604](https://github.com/poseidon/typhoon/pull/604))
+* Inline Kubelet service on Container Linux nodes ([#606](https://github.com/poseidon/typhoon/pull/606))
+* Disable unused Kubelet `127.0.0.1:10248` healthz listener ([#607](https://github.com/poseidon/typhoon/pull/607))
+* Enable kube-proxy metrics and allow Prometheus scrapes
+  * Allow TCP/10249 traffic with worker node sources
+
+#### AWS
+
+* Update Fedora CoreOS AMI filter for fedora-coreos-31 ([#620](https://github.com/poseidon/typhoon/pull/620))
+
+#### Google
+
+* Allow `terraform-provider-google` v3.0+ ([#617](https://github.com/poseidon/typhoon/pull/617))
+  * Only enforce `v2.19+` to ease migration, as no v3.x features are used
+
+#### Addons
+
+* Update Prometheus from v2.14.0 to [v2.15.2](https://github.com/prometheus/prometheus/releases/tag/v2.15.2)
+  * Add discovery for kube-proxy service endpoints
+* Update kube-state-metrics from v1.8.0 to v1.9.2
+* Reduce node-exporter DaemonSet tolerations ([#614](https://github.com/poseidon/typhoon/pull/614))
+* Update Grafana from v6.5.1 to v6.5.3
+
 ## v1.17.0

 * Kubernetes [v1.17.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.17.md#v1170)
@ -226,7 +274,7 @@ Notable changes between versions.
 * Require `terraform-provider-azurerm` v1.27+ to support Terraform v0.12 (action required)
 * Avoid unneeded rotations of Regular priority virtual machine scale sets
  * Azure only allows `eviction_policy` to be set for Low priority VMs. Supporting Low priority VMs meant when Regular VMs were used, each `terraform apply` rolled workers, to set eviction_policy to null.
-  * Terraform v0.12 nullable variables fix the issue so plan does not produce a diff. 
+  * Terraform v0.12 nullable variables fix the issue so plan does not produce a diff.

 #### Bare-Metal

@ -281,7 +329,7 @@ Notable changes between versions.
 * Update Grafana from v6.1.6 to v6.2.1

 ## v1.14.2
- 
+
 * Kubernetes [v1.14.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#v1142)
 * Update etcd from v3.3.12 to [v3.3.13](https://github.com/etcd-io/etcd/releases/tag/v3.3.13)
 * Upgrade Calico from v3.6.1 to [v3.7.2](https://docs.projectcalico.org/v3.7/release-notes/)
@ -352,7 +400,7 @@ Notable changes between versions.

 * Add ability to load balance TCP/UDP applications ([#442](https://github.com/poseidon/typhoon/pull/442))
  * Add worker instances to a target pool, output as `worker_target_pool`
-  * Health check for workers with Ingress controllers. Forward rules don't support differing internal/external ports, but some Ingress controllers support TCP/UDP proxy as a workaround 
+  * Health check for workers with Ingress controllers. Forward rules don't support differing internal/external ports, but some Ingress controllers support TCP/UDP proxy as a workaround
 * Remove Haswell minimum CPU platform requirement ([#439](https://github.com/poseidon/typhoon/pull/439))
  * Google Cloud API implements `min_cpu_platform` to mean "use exactly this CPU". Revert [#405](https://github.com/poseidon/typhoon/pull/405) added in v1.13.4.
  * Fix error creating clusters in new regions without Haswell (e.g. europe-west2) ([#438](https://github.com/poseidon/typhoon/issues/438))
@ -537,7 +585,7 @@ Notable changes between versions.
 * Update Calico from v3.3.0 to [v3.3.1](https://docs.projectcalico.org/v3.3/releases/)
  * Disable Felix usage reporting by default ([#345](https://github.com/poseidon/typhoon/pull/345))
 * Improve flannel manifests
-  * [Rename](https://github.com/poseidon/terraform-render-bootkube/commit/d045a8e6b8eccfbb9d69bb51953b5a93d23f67f7) `kube-flannel` DaemonSet to `flannel` and `kube-flannel-cfg` ConfigMap to `flannel-config` 
+  * [Rename](https://github.com/poseidon/terraform-render-bootkube/commit/d045a8e6b8eccfbb9d69bb51953b5a93d23f67f7) `kube-flannel` DaemonSet to `flannel` and `kube-flannel-cfg` ConfigMap to `flannel-config`
  * [Drop](https://github.com/poseidon/terraform-render-bootkube/commit/39f9afb3360ec642e5b98457c8bd07eda35b6c96) unused mounts and add a CPU resource request
 * Update CoreDNS from v1.2.4 to [v1.2.6](https://coredns.io/2018/11/05/coredns-1.2.6-release/)
  * Enable CoreDNS `loop` and `loadbalance` plugins ([#340](https://github.com/poseidon/typhoon/pull/340))
@ -699,7 +747,7 @@ Notable changes between versions.
 * Force apiserver to stop listening on `127.0.0.1:8080`
 * Replace `kube-dns` with [CoreDNS](https://coredns.io/) ([#261](https://github.com/poseidon/typhoon/pull/261))
  * Edit the `coredns` ConfigMap to [customize](https://coredns.io/plugins/)
-  * CoreDNS doesn't use a resizer. For large clusters, scaling may be required. 
+  * CoreDNS doesn't use a resizer. For large clusters, scaling may be required.

 #### AWS

@ -744,7 +792,7 @@ Notable changes between versions.

 * Switch `kube-apiserver` port from 443 to 6443 ([#248](https://github.com/poseidon/typhoon/pull/248))
  * Users who exposed kube-apiserver on a WAN via their router/load-balancer will need to adjust its configuration (e.g. DNAT 6443). Most apiservers are on a LAN (internal, VPN-only, etc) so if you didn't specially configure network gear for 443, no change is needed. (possible action required)
-* Fix possible deadlock when provisioning clusters larger than 10 nodes ([#244](https://github.com/poseidon/typhoon/pull/244)) 
+* Fix possible deadlock when provisioning clusters larger than 10 nodes ([#244](https://github.com/poseidon/typhoon/pull/244))

 #### DigitalOcean

@ -812,7 +860,7 @@ Notable changes between versions.
  * Please change values stable, beta, or alpha to coreos-stable, coreos-beta, coreos-alpha (**action required!**)
 * Replace `container_linux_version` variable with `os_version`
 * Add `network_ip_autodetection_method` variable for Calico host IPv4 address detection
-  * Use Calico's default "first-found" to support single NIC and bonded NIC nodes 
+  * Use Calico's default "first-found" to support single NIC and bonded NIC nodes
  * Allow [alternative](https://docs.projectcalico.org/v3.1/reference/node/configuration#ip-autodetection-methods) methods for multi NIC nodes, like can-reach=IP or interface=REGEX
 * Deprecate `container_linux_oem` variable

@ -845,7 +893,7 @@ Notable changes between versions.
 #### Google Cloud

 * Add support for multi-controller clusters (i.e. multi-master) ([#54](https://github.com/poseidon/typhoon/issues/54), [#190](https://github.com/poseidon/typhoon/pull/190))
-  * Switch from Google Cloud network load balancer to a TCP proxy load balancer. Avoid a [bug](https://issuetracker.google.com/issues/67366622) in Google network load balancers that limited clusters to only bootstrapping one controller node. 
+  * Switch from Google Cloud network load balancer to a TCP proxy load balancer. Avoid a [bug](https://issuetracker.google.com/issues/67366622) in Google network load balancers that limited clusters to only bootstrapping one controller node.
  * Add TCP health check for apiserver pods on controllers. Replace kubelet check approximation.

 #### Addons
@ -1076,7 +1124,7 @@ Notable changes between versions.
  * Container Linux stable, beta, and alpha now provide Docker 17.09 (instead
  of 1.12)
  * Older clusters (with CLUO addon) auto-update Container Linux version to begin using Docker 17.09
-* Fix race where `etcd-member.service` could fail to resolve peers ([#69](https://github.com/poseidon/typhoon/pull/69)) 
+* Fix race where `etcd-member.service` could fail to resolve peers ([#69](https://github.com/poseidon/typhoon/pull/69))
 * Add optional `cluster_domain_suffix` variable (#74)
 * Use kubernetes-incubator/bootkube v0.9.1

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@ -29,12 +29,12 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |

-A preview of Typhoon for [Fedora CoreOS](https://getfedora.org/coreos/) is available for testing.
+Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/) in alpha.

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | preview |
-| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | preview |
+| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | alpha |
+| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | alpha |

 ## Documentation

@ -48,7 +48,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.2"

  # Google Cloud
  cluster_name  = "yavin"
@ -58,7 +58,7 @@ module "yavin" {

  # configuration
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-  
+
  # optional
  worker_count = 2
  worker_preemptible = true
@ -87,9 +87,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.2
 ```

 List the pods.
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -23,7 +23,7 @@ spec:
    spec:
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:6.5.1
+          image: docker.io/grafana/grafana:6.5.3
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
@ -76,6 +76,6 @@ spec:
              - NET_BIND_SERVICE
              drop:
              - ALL
-            runAsUser: 33 # www-data
+            runAsUser: 101 # www-data
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
@ -76,6 +76,6 @@ spec:
              - NET_BIND_SERVICE
              drop:
              - ALL
-            runAsUser: 33 # www-data
+            runAsUser: 101 # www-data
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
@ -73,7 +73,7 @@ spec:
              - NET_BIND_SERVICE
              drop:
              - ALL
-            runAsUser: 33 # www-data
+            runAsUser: 101 # www-data
      restartPolicy: Always
      terminationGracePeriodSeconds: 300

--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
@ -76,6 +76,6 @@ spec:
              - NET_BIND_SERVICE
              drop:
              - ALL
-            runAsUser: 33 # www-data
+            runAsUser: 101 # www-data
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
@ -76,6 +76,6 @@ spec:
              - NET_BIND_SERVICE
              drop:
              - ALL
-            runAsUser: 33 # www-data
+            runAsUser: 101 # www-data
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -20,7 +20,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.14.0
+          image: quay.io/prometheus/prometheus:v2.15.2
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/discovery/kube-controller-manager.yaml
+++ b/addons/prometheus/discovery/kube-controller-manager.yaml
@ -1,3 +1,4 @@
+# Allow Prometheus to scrape service endpoints
 apiVersion: v1
 kind: Service
 metadata:
@ -7,7 +8,6 @@ metadata:
    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
-  # service is created to allow prometheus to scrape endpoints
  clusterIP: None
  selector:
    k8s-app: kube-controller-manager
--- a/addons/prometheus/discovery/kube-proxy.yaml
+++ b/addons/prometheus/discovery/kube-proxy.yaml
@ -0,0 +1,19 @@
+# Allow Prometheus to scrape service endpoints
+apiVersion: v1
+kind: Service
+metadata:
+  name: kube-proxy
+  namespace: kube-system
+  annotations:
+    prometheus.io/scrape: 'true'
+    prometheus.io/port: '10249'
+spec:
+  type: ClusterIP
+  clusterIP: None
+  selector:
+    k8s-app: kube-proxy
+  ports:
+    - name: metrics
+      protocol: TCP
+      port: 10249
+      targetPort: 10249
--- a/addons/prometheus/discovery/kube-scheduler.yaml
+++ b/addons/prometheus/discovery/kube-scheduler.yaml
@ -1,3 +1,4 @@
+# Allow Prometheus to scrape service endpoints
 apiVersion: v1
 kind: Service
 metadata:
@ -7,7 +8,6 @@ metadata:
    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
-  # service is created to allow prometheus to scrape endpoints
  clusterIP: None
  selector:
    k8s-app: kube-scheduler
--- a/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
@ -74,6 +74,7 @@ rules:
  - storage.k8s.io
  resources:
  - storageclasses
+  - volumeattachments
  verbs:
  - list
  - watch
@ -84,4 +85,19 @@ rules:
  verbs:
  - list
  - watch
+- apiGroups:
+  - admissionregistration.k8s.io
+  resources:
+  - mutatingwebhookconfigurations
+  - validatingwebhookconfigurations
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - networking.k8s.io
+  resources:
+  - networkpolicies
+  verbs:
+  - list
+  - watch

--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -24,7 +24,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.8.0
+        image: quay.io/coreos/kube-state-metrics:v1.9.3
        ports:
          - name: metrics
            containerPort: 8080
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -57,7 +57,9 @@ spec:
            mountPath: /host/root
            readOnly: true
      tolerations:
-        - effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+        - key: node.kubernetes.io/not-ready
          operator: Exists
      volumes:
        - name: proc
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/container-linux/kubernetes/ami.tf
+++ b/aws/container-linux/kubernetes/ami.tf
@ -4,8 +4,8 @@ locals {
  # flatcar-stable -> Flatcar Linux AMI
  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id

-  flavor  = element(split("-", var.os_image), 0)
-  channel = element(split("-", var.os_image), 1)
+  flavor  = split("-", var.os_image)[0]
+  channel = split("-", var.os_image)[1]
 }

 data "aws_ami" "coreos" {
--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -50,29 +50,47 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --insecure-options=image"
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -82,6 +100,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
@ -115,7 +134,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -130,14 +149,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/kubernetes/kubelet.env
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
    - path: /opt/bootstrap/layout
      filesystem: root
      mode: 0544
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -10,7 +10,7 @@ resource "aws_route53_record" "etcds" {
  ttl  = 300

  # private IPv4 address for etcd
-  records = [element(aws_instance.controllers.*.private_ip, count.index)]
+  records = [aws_instance.controllers.*.private_ip[count.index]]
 }

 # Controller instances
@ -24,7 +24,7 @@ resource "aws_instance" "controllers" {
  instance_type = var.controller_type

  ami       = local.ami_id
-  user_data = element(data.ct_config.controller-ignitions.*.rendered, count.index)
+  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]

  # storage
  root_block_device {
@ -36,7 +36,7 @@ resource "aws_instance" "controllers" {

  # network
  associate_public_ip_address = true
-  subnet_id                   = element(aws_subnet.public.*.id, count.index)
+  subnet_id                   = aws_subnet.public.*.id[count.index]
  vpc_security_group_ids      = [aws_security_group.controller.id]

  lifecycle {
@ -49,11 +49,8 @@ resource "aws_instance" "controllers" {

 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count = var.controller_count
-  content = element(
-    data.template_file.controller-configs.*.rendered,
-    count.index,
-  )
+  count        = var.controller_count
+  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
  snippets     = var.controller_clc_snippets
 }
@ -62,7 +59,7 @@ data "ct_config" "controller-ignitions" {
 data "template_file" "controller-configs" {
  count = var.controller_count

-  template = file("${path.module}/cl/controller.yaml.tmpl")
+  template = file("${path.module}/cl/controller.yaml")

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
--- a/aws/container-linux/kubernetes/network.tf
+++ b/aws/container-linux/kubernetes/network.tf
@ -62,6 +62,6 @@ resource "aws_route_table_association" "public" {
  count = length(data.aws_availability_zones.all.names)

  route_table_id = aws_route_table.default.id
-  subnet_id      = element(aws_subnet.public.*.id, count.index)
+  subnet_id      = aws_subnet.public.*.id[count.index]
 }

--- a/aws/container-linux/kubernetes/nlb.tf
+++ b/aws/container-linux/kubernetes/nlb.tf
@ -88,7 +88,7 @@ resource "aws_lb_target_group_attachment" "controllers" {
  count = var.controller_count

  target_group_arn = aws_lb_target_group.controllers.arn
-  target_id        = element(aws_instance.controllers.*.id, count.index)
+  target_id        = aws_instance.controllers.*.id[count.index]
  port             = 6443
 }

--- a/aws/container-linux/kubernetes/security.tf
+++ b/aws/container-linux/kubernetes/security.tf
@ -33,6 +33,28 @@ resource "aws_security_group_rule" "controller-etcd" {
  self      = true
 }

+# Allow Prometheus to scrape etcd metrics
+resource "aws_security_group_rule" "controller-etcd-metrics" {
+  security_group_id = aws_security_group.controller.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 2381
+  to_port                  = 2381
+  source_security_group_id = aws_security_group.worker.id
+}
+
+# Allow Prometheus to scrape kube-proxy
+resource "aws_security_group_rule" "kube-proxy-metrics" {
+  security_group_id = aws_security_group.controller.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10249
+  to_port                  = 10249
+  source_security_group_id = aws_security_group.worker.id
+}
+
 # Allow Prometheus to scrape kube-scheduler
 resource "aws_security_group_rule" "controller-scheduler-metrics" {
  security_group_id = aws_security_group.controller.id
@ -55,17 +77,6 @@ resource "aws_security_group_rule" "controller-manager-metrics" {
  source_security_group_id = aws_security_group.worker.id
 }

-# Allow Prometheus to scrape etcd metrics
-resource "aws_security_group_rule" "controller-etcd-metrics" {
-  security_group_id = aws_security_group.controller.id
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 2381
-  to_port                  = 2381
-  source_security_group_id = aws_security_group.worker.id
-}
-
 resource "aws_security_group_rule" "controller-vxlan" {
  count = var.networking == "flannel" ? 1 : 0

@ -281,14 +292,15 @@ resource "aws_security_group_rule" "worker-node-exporter" {
  self      = true
 }

-resource "aws_security_group_rule" "ingress-health" {
+# Allow Prometheus to scrape kube-proxy
+resource "aws_security_group_rule" "worker-kube-proxy" {
  security_group_id = aws_security_group.worker.id

-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 10254
-  to_port     = 10254
-  cidr_blocks = ["0.0.0.0/0"]
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10249
+  to_port   = 10249
+  self      = true
 }

 # Allow apiserver to access kubelets for exec, log, port-forward
@ -313,6 +325,16 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
  self      = true
 }

+resource "aws_security_group_rule" "ingress-health" {
+  security_group_id = aws_security_group.worker.id
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 10254
+  to_port     = 10254
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
 resource "aws_security_group_rule" "worker-bgp" {
  security_group_id = aws_security_group.worker.id

--- a/aws/container-linux/kubernetes/ssh.tf
+++ b/aws/container-linux/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
--- a/aws/container-linux/kubernetes/workers/ami.tf
+++ b/aws/container-linux/kubernetes/workers/ami.tf
@ -4,8 +4,8 @@ locals {
  # flatcar-stable -> Flatcar Linux AMI
  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id

-  flavor  = element(split("-", var.os_image), 0)
-  channel = element(split("-", var.os_image), 1)
+  flavor  = split("-", var.os_image)[0]
+  channel = split("-", var.os_image)[1]
 }

 data "aws_ami" "coreos" {
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -25,29 +25,47 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --insecure-options=image"
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -57,6 +75,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
@ -92,14 +111,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/kubernetes/kubelet.env
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -117,7 +128,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            -- \
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -78,7 +78,7 @@ data "ct_config" "worker-ignition" {

 # Worker Container Linux config
 data "template_file" "worker-config" {
-  template = file("${path.module}/cl/worker.yaml.tmpl")
+  template = file("${path.module}/cl/worker.yaml")

  vars = {
    kubeconfig             = indent(10, var.kubeconfig)
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -15,9 +15,9 @@ data "aws_ami" "fedora-coreos" {

  filter {
    name   = "name"
-    values = ["fedora-coreos-30.*.*-hvm"]
+    values = ["fedora-coreos-31.*.*.*-hvm"]
  }

  # try to filter out dev images (AWS filters can't)
-  name_regex = "^fedora-coreos-30.[0-9]*.[0-9]*-hvm*"
+  name_regex = "^fedora-coreos-31.[0-9]*.[0-9]*.[0-9]*-hvm*"
 }
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -77,10 +77,9 @@ systemd:
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
-          --volume /var/run:/var/run \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          k8s.gcr.io/hyperkube:v1.17.0 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -92,6 +91,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
@ -123,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            k8s.gcr.io/hyperkube:v1.17.0
+            k8s.gcr.io/hyperkube:v1.17.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/aws/fedora-coreos/kubernetes/security.tf
+++ b/aws/fedora-coreos/kubernetes/security.tf
@ -44,6 +44,17 @@ resource "aws_security_group_rule" "controller-etcd-metrics" {
  source_security_group_id = aws_security_group.worker.id
 }

+# Allow Prometheus to scrape kube-proxy
+resource "aws_security_group_rule" "kube-proxy-metrics" {
+  security_group_id = aws_security_group.controller.id
+
+  type                     = "ingress"
+  protocol                 = "tcp"
+  from_port                = 10249
+  to_port                  = 10249
+  source_security_group_id = aws_security_group.worker.id
+}
+
 # Allow Prometheus to scrape kube-scheduler
 resource "aws_security_group_rule" "controller-scheduler-metrics" {
  security_group_id = aws_security_group.controller.id
@ -281,14 +292,15 @@ resource "aws_security_group_rule" "worker-node-exporter" {
  self      = true
 }

-resource "aws_security_group_rule" "ingress-health" {
+# Allow Prometheus to scrape kube-proxy
+resource "aws_security_group_rule" "worker-kube-proxy" {
  security_group_id = aws_security_group.worker.id

-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 10254
-  to_port     = 10254
-  cidr_blocks = ["0.0.0.0/0"]
+  type      = "ingress"
+  protocol  = "tcp"
+  from_port = 10249
+  to_port   = 10249
+  self      = true
 }

 # Allow apiserver to access kubelets for exec, log, port-forward
@ -313,6 +325,16 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
  self      = true
 }

+resource "aws_security_group_rule" "ingress-health" {
+  security_group_id = aws_security_group.worker.id
+
+  type        = "ingress"
+  protocol    = "tcp"
+  from_port   = 10254
+  to_port     = 10254
+  cidr_blocks = ["0.0.0.0/0"]
+}
+
 resource "aws_security_group_rule" "worker-bgp" {
  security_group_id = aws_security_group.worker.id

--- a/aws/fedora-coreos/kubernetes/ssh.tf
+++ b/aws/fedora-coreos/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
@ -21,7 +21,7 @@ resource "null_resource" "copy-controller-secrets" {
    user    = "core"
    timeout = "15m"
  }
-  
+
  provisioner "file" {
    content     = join("\n", local.assets_bundle)
    destination = "$HOME/assets"
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -15,9 +15,9 @@ data "aws_ami" "fedora-coreos" {

  filter {
    name   = "name"
-    values = ["fedora-coreos-30.*.*-hvm"]
+    values = ["fedora-coreos-31.*.*.*-hvm"]
  }

  # try to filter out dev images (AWS filters can't)
-  name_regex = "^fedora-coreos-30.[0-9]*.[0-9]*-hvm*"
+  name_regex = "^fedora-coreos-31.[0-9]*.[0-9]*.[0-9]*-hvm*"
 }
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -47,10 +47,9 @@ systemd:
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
-          --volume /var/run:/var/run \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          k8s.gcr.io/hyperkube:v1.17.0 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -62,6 +61,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
--- a/aws/ignore/.gitkeep
+++ b/aws/ignore/.gitkeep
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/azure/container-linux/kubernetes/bootstrap.tf
+++ b/azure/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -50,29 +50,46 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --hosts-entry=host \
-          --insecure-options=image"
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -81,14 +98,15 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -114,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -129,14 +147,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/kubernetes/kubelet.env
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
    - path: /opt/bootstrap/layout
      filesystem: root
      mode: 0544
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -11,16 +11,13 @@ resource "azurerm_dns_a_record" "etcds" {
  ttl  = 300

  # private IPv4 address for etcd
-  records = [element(
-    azurerm_network_interface.controllers.*.private_ip_address,
-    count.index,
-  )]
+  records = [azurerm_network_interface.controllers.*.private_ip_address[count.index]]
 }

 locals {
  # Channel for a Container Linux derivative
  # coreos-stable -> Container Linux Stable
-  channel = element(split("-", var.os_image), 1)
+  channel = split("-", var.os_image)[1]
 }

 # Controller availability set to spread controllers
@ -63,12 +60,12 @@ resource "azurerm_virtual_machine" "controllers" {
  }

  # network
-  network_interface_ids = [element(azurerm_network_interface.controllers.*.id, count.index)]
+  network_interface_ids = [azurerm_network_interface.controllers.*.id[count.index]]

  os_profile {
    computer_name  = "${var.cluster_name}-controller-${count.index}"
    admin_username = "core"
-    custom_data    = element(data.ct_config.controller-ignitions.*.rendered, count.index)
+    custom_data    = data.ct_config.controller-ignitions.*.rendered[count.index]
  }

  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@ -108,7 +105,7 @@ resource "azurerm_network_interface" "controllers" {
    private_ip_address_allocation = "dynamic"

    # public IPv4
-    public_ip_address_id = element(azurerm_public_ip.controllers.*.id, count.index)
+    public_ip_address_id = azurerm_public_ip.controllers.*.id[count.index]
  }
 }

@ -134,11 +131,8 @@ resource "azurerm_public_ip" "controllers" {

 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count = var.controller_count
-  content = element(
-    data.template_file.controller-configs.*.rendered,
-    count.index,
-  )
+  count        = var.controller_count
+  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
  snippets     = var.controller_clc_snippets
 }
@ -147,7 +141,7 @@ data "ct_config" "controller-ignitions" {
 data "template_file" "controller-configs" {
  count = var.controller_count

-  template = file("${path.module}/cl/controller.yaml.tmpl")
+  template = file("${path.module}/cl/controller.yaml")

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
--- a/azure/container-linux/kubernetes/security.tf
+++ b/azure/container-linux/kubernetes/security.tf
@ -53,13 +53,29 @@ resource "azurerm_network_security_rule" "controller-etcd-metrics" {
  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

+# Allow Prometheus to scrape kube-proxy metrics
+resource "azurerm_network_security_rule" "controller-kube-proxy" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kube-proxy-metrics"
+  network_security_group_name = azurerm_network_security_group.controller.name
+  priority                    = "2011"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10249"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+}
+
 # Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
 resource "azurerm_network_security_rule" "controller-kube-metrics" {
  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-kube-metrics"
  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2011"
+  priority                    = "2012"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Tcp"
@ -251,6 +267,22 @@ resource "azurerm_network_security_rule" "worker-node-exporter" {
  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }

+# Allow Prometheus to scrape kube-proxy
+resource "azurerm_network_security_rule" "worker-kube-proxy" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name                        = "allow-kube-proxy"
+  network_security_group_name = azurerm_network_security_group.worker.name
+  priority                    = "2024"
+  access                      = "Allow"
+  direction                   = "Inbound"
+  protocol                    = "Tcp"
+  source_port_range           = "*"
+  destination_port_range      = "10249"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+}
+
 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "worker-kubelet" {
  resource_group_name = azurerm_resource_group.cluster.name
--- a/azure/container-linux/kubernetes/ssh.tf
+++ b/azure/container-linux/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
@ -22,7 +22,7 @@ resource "null_resource" "copy-controller-secrets" {
    user    = "core"
    timeout = "15m"
  }
-  
+
  provisioner "file" {
    content     = join("\n", local.assets_bundle)
    destination = "$HOME/assets"
@ -45,7 +45,7 @@ resource "null_resource" "bootstrap" {

  connection {
    type    = "ssh"
-    host    = element(azurerm_public_ip.controllers.*.ip_address, 0)
+    host    = azurerm_public_ip.controllers.*.ip_address[0]
    user    = "core"
    timeout = "15m"
  }
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -25,28 +25,46 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --insecure-options=image"
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -55,6 +73,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
@ -90,14 +109,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/kubernetes/kubelet.env
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -115,7 +126,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            -- \
--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -1,7 +1,7 @@
 locals {
  # Channel for a Container Linux derivative
  # coreos-stable -> Container Linux Stable
-  channel = element(split("-", var.os_image), 1)
+  channel = split("-", var.os_image)[1]
 }

 # Workers scale set
@ -104,7 +104,7 @@ data "ct_config" "worker-ignition" {

 # Worker Container Linux configs
 data "template_file" "worker-config" {
-  template = file("${path.module}/cl/worker.yaml.tmpl")
+  template = file("${path.module}/cl/worker.yaml")

  vars = {
    kubeconfig             = indent(10, var.kubeconfig)
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/container-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -58,33 +58,51 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
-          --mount volume=iscsiconf,target=/etc/iscsi/ \
-          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
-          --mount volume=iscsiadm,target=/sbin/iscsiadm \
-          --insecure-options=image"
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          --volume etc-iscsi,kind=host,source=/etc/iscsi \
+          --mount volume=etc-iscsi,target=/etc/iscsi \
+          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -94,6 +112,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
@ -128,7 +147,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -136,15 +155,10 @@ systemd:
        [Install]
        WantedBy=multi-user.target
 storage:
-  files:
-    - path: /etc/kubernetes/kubelet.env
+  directories:
+    - path: /etc/kubernetes
      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
+  files:
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/kubernetes/cl/install.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/install.yaml.tmpl
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -33,33 +33,51 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume iscsiconf,kind=host,source=/etc/iscsi/ \
-          --mount volume=iscsiconf,target=/etc/iscsi/ \
-          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
-          --mount volume=iscsiadm,target=/sbin/iscsiadm \
-          --insecure-options=image"
        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          --volume etc-iscsi,kind=host,source=/etc/iscsi \
+          --mount volume=etc-iscsi,target=/etc/iscsi \
+          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
+          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -69,6 +87,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
@ -84,15 +103,10 @@ systemd:
        WantedBy=multi-user.target

 storage:
-  files:
-    - path: /etc/kubernetes/kubelet.env
+  directories:
+    - path: /etc/kubernetes
      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
+  files:
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -31,7 +31,7 @@ resource "matchbox_profile" "container-linux-install" {
 data "template_file" "container-linux-install-configs" {
  count = length(var.controllers) + length(var.workers)

-  template = file("${path.module}/cl/install.yaml.tmpl")
+  template = file("${path.module}/cl/install.yaml")

  vars = {
    os_flavor          = local.flavor
@ -72,7 +72,7 @@ resource "matchbox_profile" "cached-container-linux-install" {
 data "template_file" "cached-container-linux-install-configs" {
  count = length(var.controllers) + length(var.workers)

-  template = file("${path.module}/cl/install.yaml.tmpl")
+  template = file("${path.module}/cl/install.yaml")

  vars = {
    os_flavor          = local.flavor
@ -150,7 +150,7 @@ data "ct_config" "controller-ignitions" {
 data "template_file" "controller-configs" {
  count = length(var.controllers)

-  template = file("${path.module}/cl/controller.yaml.tmpl")
+  template = file("${path.module}/cl/controller.yaml")

  vars = {
    domain_name            = var.controllers.*.domain[count.index]
@ -180,7 +180,7 @@ data "ct_config" "worker-ignitions" {
 data "template_file" "worker-configs" {
  count = length(var.workers)

-  template = file("${path.module}/cl/worker.yaml.tmpl")
+  template = file("${path.module}/cl/worker.yaml")

  vars = {
    domain_name            = var.workers.*.domain[count.index]
--- a/bare-metal/container-linux/kubernetes/ssh.tf
+++ b/bare-metal/container-linux/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -76,12 +76,11 @@ systemd:
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
-          --volume /var/run:/var/run \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          k8s.gcr.io/hyperkube:v1.17.0 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -93,6 +92,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
@ -134,7 +134,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            k8s.gcr.io/hyperkube:v1.17.0
+            k8s.gcr.io/hyperkube:v1.17.2
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -46,12 +46,11 @@ systemd:
          --volume /var/lib/docker:/var/lib/docker \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
-          --volume /var/run:/var/run \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          k8s.gcr.io/hyperkube:v1.17.0 kubelet \
+          k8s.gcr.io/hyperkube:v1.17.2 kubelet \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -63,6 +62,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --hostname-override=${domain_name} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -1,6 +1,6 @@
 locals {
-  remote_kernel = "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-installer-kernel-x86_64"
-  remote_initrd = "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-installer-initramfs.x86_64.img"
+  remote_kernel = "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-live-kernel-x86_64"
+  remote_initrd = "https://builds.coreos.fedoraproject.org/prod/streams/${var.os_stream}/builds/${var.os_version}/x86_64/fedora-coreos-${var.os_version}-live-initramfs.x86_64.img"
  remote_args = [
    "ip=dhcp",
    "rd.neednet=1",
@ -10,8 +10,8 @@ locals {
    "coreos.inst.install_dev=${var.install_disk}"
  ]

-  cached_kernel = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-installer-kernel-x86_64"
-  cached_initrd = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-installer-initramfs.x86_64.img"
+  cached_kernel = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-kernel-x86_64"
+  cached_initrd = "/assets/fedora-coreos/fedora-coreos-${var.os_version}-live-initramfs.x86_64.img"
  cached_args = [
    "ip=dhcp",
    "rd.neednet=1",
--- a/bare-metal/fedora-coreos/kubernetes/ssh.tf
+++ b/bare-metal/fedora-coreos/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
--- a/bare-metal/ignore/.gitkeep
+++ b/bare-metal/ignore/.gitkeep
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/digital-ocean/container-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -60,29 +60,47 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
        EnvironmentFile=/run/metadata/coreos
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --insecure-options=image"
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -91,6 +109,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
@ -125,7 +144,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -133,15 +152,10 @@ systemd:
        [Install]
        WantedBy=multi-user.target
 storage:
-  files:
-    - path: /etc/kubernetes/kubelet.env
+  directories:
+    - path: /etc/kubernetes
      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
+  files:
    - path: /opt/bootstrap/layout
      filesystem: root
      mode: 0544
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -35,29 +35,47 @@ systemd:
        After=coreos-metadata.service
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
        EnvironmentFile=/run/metadata/coreos
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --insecure-options=image"
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -66,6 +84,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
@ -92,15 +111,10 @@ systemd:
        [Install]
        WantedBy=multi-user.target
 storage:
-  files:
-    - path: /etc/kubernetes/kubelet.env
+  directories:
+    - path: /etc/kubernetes
      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
+  files:
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -118,7 +132,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            -- \
--- a/digital-ocean/container-linux/kubernetes/controllers.tf
+++ b/digital-ocean/container-linux/kubernetes/controllers.tf
@ -11,7 +11,7 @@ resource "digitalocean_record" "controllers" {
  ttl  = 300

  # IPv4 addresses of controllers
-  value = element(digitalocean_droplet.controllers.*.ipv4_address, count.index)
+  value = digitalocean_droplet.controllers.*.ipv4_address[count.index]
 }

 # Discrete DNS records for each controller's private IPv4 for etcd usage
@ -27,7 +27,7 @@ resource "digitalocean_record" "etcds" {
  ttl  = 300

  # private IPv4 address for etcd
-  value = element(digitalocean_droplet.controllers.*.ipv4_address_private, count.index)
+  value = digitalocean_droplet.controllers.*.ipv4_address_private[count.index]
 }

 # Controller droplet instances
@ -44,7 +44,7 @@ resource "digitalocean_droplet" "controllers" {
  ipv6               = true
  private_networking = true

-  user_data = element(data.ct_config.controller-ignitions.*.rendered, count.index)
+  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]
  ssh_keys  = var.ssh_fingerprints

  tags = [
@ -64,7 +64,7 @@ resource "digitalocean_tag" "controllers" {
 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
  count        = var.controller_count
-  content      = element(data.template_file.controller-configs.*.rendered, count.index)
+  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
  snippets     = var.controller_clc_snippets
 }
@ -73,7 +73,7 @@ data "ct_config" "controller-ignitions" {
 data "template_file" "controller-configs" {
  count = var.controller_count

-  template = file("${path.module}/cl/controller.yaml.tmpl")
+  template = file("${path.module}/cl/controller.yaml")

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
--- a/digital-ocean/container-linux/kubernetes/network.tf
+++ b/digital-ocean/container-linux/kubernetes/network.tf
@ -16,12 +16,20 @@ resource "digitalocean_firewall" "rules" {
    source_tags = [digitalocean_tag.controllers.name, digitalocean_tag.workers.name]
  }

+  # Allow Prometheus to scrape node-exporter
  inbound_rule {
    protocol    = "tcp"
    port_range  = "9100"
    source_tags = [digitalocean_tag.workers.name]
  }

+  # Allow Prometheus to scrape kube-proxy
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "10249"
+    source_tags = [digitalocean_tag.workers.name]
+  }
+
  inbound_rule {
    protocol    = "tcp"
    port_range  = "10250"
--- a/digital-ocean/container-linux/kubernetes/outputs.tf
+++ b/digital-ocean/container-linux/kubernetes/outputs.tf
@ -27,6 +27,12 @@ output "workers_ipv6" {
  value = digitalocean_droplet.workers.*.ipv6_address
 }

+# Outputs for worker pools
+
+output "kubeconfig" {
+  value = module.bootstrap.kubeconfig-kubelet
+}
+
 # Outputs for custom firewalls

 output "controller_tag" {
--- a/digital-ocean/container-linux/kubernetes/ssh.tf
+++ b/digital-ocean/container-linux/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
--- a/digital-ocean/container-linux/kubernetes/workers.tf
+++ b/digital-ocean/container-linux/kubernetes/workers.tf
@ -8,7 +8,7 @@ resource "digitalocean_record" "workers-record-a" {
  name  = "${var.cluster_name}-workers"
  type  = "A"
  ttl   = 300
-  value = element(digitalocean_droplet.workers.*.ipv4_address, count.index)
+  value = digitalocean_droplet.workers.*.ipv4_address[count.index]
 }

 resource "digitalocean_record" "workers-record-aaaa" {
@ -20,7 +20,7 @@ resource "digitalocean_record" "workers-record-aaaa" {
  name  = "${var.cluster_name}-workers"
  type  = "AAAA"
  ttl   = 300
-  value = element(digitalocean_droplet.workers.*.ipv6_address, count.index)
+  value = digitalocean_droplet.workers.*.ipv6_address[count.index]
 }

 # Worker droplet instances
@ -63,7 +63,7 @@ data "ct_config" "worker-ignition" {

 # Worker Container Linux config
 data "template_file" "worker-config" {
-  template = file("${path.module}/cl/worker.yaml.tmpl")
+  template = file("${path.module}/cl/worker.yaml")

  vars = {
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -79,7 +79,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define

 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.17.2"
  
  # Azure
  region                  = module.ramius.region
@ -145,7 +145,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.17.2"

  # Google Cloud
  region       = "europe-west2"
@ -176,11 +176,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.17.0
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.17.0
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.17.0
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.17.0
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.17.0
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.17.2
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.17.2
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.17.2
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.17.2
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.17.2
 ```

 ### Variables
--- a/docs/announce.md
+++ b/docs/announce.md
@ -1,5 +1,15 @@
 # Announce <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo-small.png">

+## Jan 23rd, 2020
+
+Typhoon for Fedora CoreOS promoted to alpha!
+
+Last summer, Typhoon released the first preview of Kubernetes on Fedora CoreOS for bare-metal and AWS, developing many ideas and patterns from Typhoon for Container Linux and Fedora Atomic. Since then, Typhoon for Fedora CoreOS has evolved and gained features alongside Typhoon, while Fedora CoreOS itself has evolved and improved too.
+
+Fedora recently [announced](https://fedoramagazine.org/fedora-coreos-out-of-preview/) that Fedora CoreOS is available for general use. To align with that change and to better indicate the maturing status, Typhoon for Fedora CoreOS has been promoted to alpha. Many thanks to folks who have worked to make this possbile!
+
+About: For newcomers, Typhoon is a minimal and free (cost and freedom) Kubernetes distribution providing upstream Kubernetes, declarative configuration via Terraform, and support for AWS, Azure, Google Cloud, DigitalOcean, and bare-metal. It is run by former CoreOS engineer [@dghubble](https://twitter.com/dghubble) to power his clusters, with freedom [motivations](https://typhoon.psdn.io/#motivation).
+
 ## Jul 18, 2019

 Introducing a preview of Typhoon Kubernetes clusters with Fedora CoreOS!
@ -8,8 +18,6 @@ Fedora recently [announced](https://lists.fedoraproject.org/archives/list/coreos

 While Typhoon uses Container Linux (or Flatcar Linux) for stable modules, the project hasn't been a stranger to Fedora ideas, once developing a [Fedora Atomic](https://typhoon.psdn.io/announce/#april-26-2018) variant in 2018. That makes the Fedora CoreOS fushion both exciting and familiar. Typhoon with Fedora CoreOS uses Ignition v3 for provisioning, uses rpm-ostree for layering and updates, tries swapping system containers for podman, and brings SELinux enforcement ([table](https://typhoon.psdn.io/architecture/operating-systems/)). This is an early preview (don't go to prod), but do try it out and help identify and solve issues (getting started links above).

-About: For newcomers, Typhoon is a minimal and free (cost and freedom) Kubernetes distribution providing upstream Kubernetes, declarative configuration via Terraform, and support for AWS, Azure, Google Cloud, DigitalOcean, and bare-metal. It is run by former CoreOS engineer [@dghubble](https://twitter.com/dghubble) to power his clusters with freedom [motivations](https://typhoon.psdn.io/#motivation).
-
 ## March 27, 2019

 Last April, Typhoon [introduced](#april-26-2018) alpha support for creating Kubernetes clusters with Fedora Atomic on AWS, Google Cloud, DigitalOcean, and bare-metal. Fedora Atomic shared many of Container Linux's aims for a container-optimized operating system, introduced novel ideas, and provided technical diversification for an uncertain future. However, Project Atomic efforts were merged into Fedora CoreOS and future Fedora Atomic releases are [not expected](http://www.projectatomic.io/blog/2018/06/welcome-to-fedora-coreos/). *Typhoon modules for Fedora Atomic will not be updated much beyond Kubernetes v1.13*. They may later be removed.
@ -46,7 +54,7 @@ Typhoon for Fedora Atomic reflects many of the same principles that created Typh

 Meanwhile, Fedora Atomic adds some promising new low-level technologies:

-* [ostree](https://github.com/ostreedev/ostree) & [rpm-ostree](https://github.com/projectatomic/rpm-ostree) - a hybrid, layered, image and package system that lets you perform atomic updates and rollbacks, layer on packages, "rebase" your system, or manage a remote tree repo. See Dusty Mabe's great [intro](https://dustymabe.com/2017/09/01/atomic-host-101-lab-part-3-rebase-upgrade-rollback/). 
+* [ostree](https://github.com/ostreedev/ostree) & [rpm-ostree](https://github.com/projectatomic/rpm-ostree) - a hybrid, layered, image and package system that lets you perform atomic updates and rollbacks, layer on packages, "rebase" your system, or manage a remote tree repo. See Dusty Mabe's great [intro](https://dustymabe.com/2017/09/01/atomic-host-101-lab-part-3-rebase-upgrade-rollback/).

 * [system containers](http://www.projectatomic.io/blog/2016/09/intro-to-system-containers/) - OCI container images that embed systemd and runc metadata for starting low-level host services before container runtimes are ready. Typhoon uses system containers under runc for `etcd`, `kubelet`, and `bootkube` on Fedora Atomic (instead of rkt-fly).

--- a/docs/cl/aws.md
+++ b/docs/cl/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.17.0 cluster on AWS with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.2 cluster on AWS with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.17.2"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.17.0
-ip-10-0-26-65  Ready   <none>  10m  v1.17.0
-ip-10-0-41-21  Ready   <none>  10m  v1.17.0
+ip-10-0-3-155  Ready   <none>  10m  v1.17.2
+ip-10-0-26-65  Ready   <none>  10m  v1.17.2
+ip-10-0-41-21  Ready   <none>  10m  v1.17.2
 ```

 List the pods.
--- a/docs/cl/azure.md
+++ b/docs/cl/azure.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.

-In this tutorial, we'll create a Kubernetes v1.17.0 cluster on Azure with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.2 cluster on Azure with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -66,7 +66,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.17.2"

  # Azure
  cluster_name   = "ramius"
@ -140,9 +140,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.17.0
-ramius-worker-000001  Ready   <none>  25m  v1.17.0
-ramius-worker-000002  Ready   <none>  24m  v1.17.0
+ramius-controller-0   Ready   <none>  24m  v1.17.2
+ramius-worker-000001  Ready   <none>  25m  v1.17.2
+ramius-worker-000002  Ready   <none>  24m  v1.17.2
 ```

 List the pods.
--- a/docs/cl/bare-metal.md
+++ b/docs/cl/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.17.0 cluster on bare-metal with Container Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.17.2 cluster on bare-metal with Container Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.2"
  
  # bare-metal
  cluster_name            = "mercury"
@ -264,9 +264,9 @@ Apply complete! Resources: 55 added, 0 changed, 0 destroyed.
 To watch the install to disk (until machines reboot from disk), SSH to port 2222.

 ```
-# before v1.17.0
+# before v1.10.1
 $ ssh debug@node1.example.com
-# after v1.17.0
+# after v1.10.1
 $ ssh -p 2222 core@node1.example.com
 ```

@ -299,9 +299,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.17.0
-node2.example.com   Ready   <none>  10m  v1.17.0
-node3.example.com   Ready   <none>  10m  v1.17.0
+node1.example.com   Ready   <none>  10m  v1.17.2
+node2.example.com   Ready   <none>  10m  v1.17.2
+node3.example.com   Ready   <none>  10m  v1.17.2
 ```

 List the pods.
--- a/docs/cl/digital-ocean.md
+++ b/docs/cl/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.17.0 cluster on DigitalOcean with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.2 cluster on DigitalOcean with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -65,7 +65,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.17.2"

  # Digital Ocean
  cluster_name = "nemo"
@ -138,9 +138,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.17.0
-10.132.115.81      Ready   <none>  10m  v1.17.0
-10.132.124.107     Ready   <none>  10m  v1.17.0
+10.132.110.130     Ready   <none>  10m  v1.17.2
+10.132.115.81      Ready   <none>  10m  v1.17.2
+10.132.124.107     Ready   <none>  10m  v1.17.2
 ```

 List the pods.
--- a/docs/cl/google-cloud.md
+++ b/docs/cl/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.17.0 cluster on Google Compute Engine with Container Linux.
+In this tutorial, we'll create a Kubernetes v1.17.2 cluster on Google Compute Engine with Container Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -49,7 +49,7 @@ Configure the Google Cloud provider to use your service account key, project-id,

 ```tf
 provider "google" {
-  version     = "2.20.0"
+  version     = "3.4.0"
  project     = "project-id"
  region      = "us-central1"
  credentials = file("~/.config/google-cloud/terraform.json")
@ -71,7 +71,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.2"

  # Google Cloud
  cluster_name  = "yavin"
@ -145,9 +145,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.2
 ```

 List the pods.
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,9 +1,9 @@
 # AWS

 !!! danger
-    Typhoon for Fedora CoreOS is an early preview! Fedora CoreOS itself is a preview! Expect bugs and design shifts. Please help both projects solve problems. Report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues). Report Typhoon issues to Typhoon.
+    Typhoon for Fedora CoreOS is an alpha. Please report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues) and Typhoon issues to Typhoon.

-In this tutorial, we'll create a Kubernetes v1.17.0 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.17.2 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -73,7 +73,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.17.2"

  # AWS
  cluster_name = "tempest"
@ -146,9 +146,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.17.0
-ip-10-0-26-65  Ready   <none>   10m  v1.17.0
-ip-10-0-41-21  Ready   <none>   10m  v1.17.0
+ip-10-0-3-155  Ready   <none>   10m  v1.17.2
+ip-10-0-26-65  Ready   <none>   10m  v1.17.2
+ip-10-0-41-21  Ready   <none>   10m  v1.17.2
 ```

 List the pods.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,9 +1,9 @@
 # Bare-Metal

 !!! danger
-    Typhoon for Fedora CoreOS is an early preview! Fedora CoreOS itself is a preview! Expect bugs and design shifts. Please help both projects solve problems. Report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues). Report Typhoon issues to Typhoon.
+    Typhoon for Fedora CoreOS is an alpha. Please report Fedora CoreOS bugs to [Fedora](https://github.com/coreos/fedora-coreos-tracker/issues) and Typhoon issues to Typhoon.

-In this tutorial, we'll network boot and provision a Kubernetes v1.17.0 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.17.2 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -30,7 +30,7 @@ Configure each machine to boot from the disk through IPMI or the BIOS menu.
 ```
 ipmitool -H node1 -U USER -P PASS chassis bootdev disk options=persistent
 ```
- 
+
 During provisioning, you'll explicitly set the boot device to `pxe` for the next boot only. Machines will install (overwrite) the operating system to disk on PXE boot and reboot into the disk install.

 !!! tip ""
@ -106,7 +106,7 @@ Read about the [many ways](https://coreos.com/matchbox/docs/latest/network-setup
    TFTP chainloading to modern boot firmware, like iPXE, avoids issues with old NICs and allows faster transfer protocols like HTTP to be used.

 !!! warning
-    Compile iPXE from [source](https://github.com/ipxe/ipxe) with support for [HTTPS downloads](https://ipxe.org/crypto). iPXE's pre-built firmware binaries do not enable this. Fedora does not provide images over HTTP.
+    Compile iPXE from [source](https://github.com/ipxe/ipxe) with support for [HTTPS downloads](https://ipxe.org/crypto). iPXE's pre-built firmware binaries do not enable this. Fedora CoreOS downloads are HTTPS-only.

 ## Terraform Setup

@ -163,14 +163,13 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.17.0"
-  
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.17.2"
+
  # bare-metal
  cluster_name            = "mercury"
  matchbox_http_endpoint  = "http://matchbox.example.com"
-  os_stream               = "testing"
-  os_version              = "30.20191002.0"
-  cached_install          = true
+  os_stream               = "stable"
+  os_version              = "31.20200113.3.1"

  # configuration
  k8s_domain_name    = "node1.example.com"
@ -293,9 +292,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.17.0
-node2.example.com   Ready   <none>  10m  v1.17.0
-node3.example.com   Ready   <none>  10m  v1.17.0
+node1.example.com   Ready   <none>  10m  v1.17.2
+node2.example.com   Ready   <none>  10m  v1.17.2
+node3.example.com   Ready   <none>  10m  v1.17.2
 ```

 List the pods.
@ -330,8 +329,8 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 |:-----|:------------|:--------|
 | cluster_name | Unique cluster name | "mercury" |
 | matchbox_http_endpoint | Matchbox HTTP read-only endpoint | "http://matchbox.example.com:port" |
-| os_stream | Fedora CoreOS release stream | "testing" |
-| os_version | Fedora CoreOS version to PXE and install | "30.20190716.1" |
+| os_stream | Fedora CoreOS release stream | "stable" |
+| os_version | Fedora CoreOS version to PXE and install | "31.20200113.3.1" |
 | k8s_domain_name | FQDN resolving to the controller(s) nodes. Workers and kubectl will communicate with this endpoint | "myk8s.example.com" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3Nz..." |
 | controllers | List of controller machine detail objects (unique name, identifying MAC address, FQDN) | `[{name="node1", mac="52:54:00:a1:9c:ae", domain="node1.example.com"}]` |
@ -345,7 +344,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | cached_install | PXE boot and install from the Matchbox `/assets` cache. Admin MUST have downloaded Fedora CoreOS images into the cache | false | true |
 | install_disk | Disk device where Fedora CoreOS should be installed | "sda" (not "/dev/sda" like Container Linux) | "sdb" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
-| network_mtu | CNI interface MTU (calico-only) | 1480 | - | 
+| network_mtu | CNI interface MTU (calico-only) | 1480 | - |
 | snippets | Map from machine names to lists of Fedora CoreOS Config snippets | {} | UNSUPPORTED |
 | network_ip_autodetection_method | Method to detect host IPv4 address (calico-only) | "first-found" | "can-reach=10.0.0.1" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
@ -29,12 +29,12 @@ Typhoon provides a Terraform Module for each supported operating system and plat
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |

-A preview of Typhoon for [Fedora CoreOS](https://getfedora.org/coreos/) is available for testing.
+Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/) in alpha.

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | preview |
-| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | preview |
+| AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | alpha |
+| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | alpha |

 ## Documentation

@ -47,7 +47,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.17.2"

  # Google Cloud
  cluster_name  = "yavin"
@ -57,7 +57,7 @@ module "yavin" {

  # configuration
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
-  
+
  # optional
  worker_count = 2
 }
@ -85,9 +85,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.17.2
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.17.2
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.17.2
 ```

 List the pods.
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -18,7 +18,7 @@ module "yavin" {
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.17.2"
  ...
 }
 ```
@ -279,15 +279,15 @@ Typhoon modules have been adapted for Terraform v0.12. Provider plugins requirem

 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.17.0 - ?       | v0.12.x             |
-| v1.10.3 - v1.17.0 | v0.11.x             |
+| v1.17.2 - ?       | v0.12.x             |
+| v1.10.3 - v1.17.2 | v0.11.x             |
 | v1.9.2 - v1.10.2  | v0.10.4+ or v0.11.x |
 | v1.7.3 - v1.9.1   | v0.10.x             |
 | v1.6.4 - v1.7.2   | v0.9.x              |

 ### New users

-New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.17.0+ without issue.
+New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.17.2+ without issue.

 ### Existing users

@ -395,7 +395,7 @@ Verify no changes are proposed and commit changes to version control. You've mig

 Alternately, continue maintaining existing clusters using Terraform v0.11.x and existing Terraform configuration directory(ies). Create new Terraform directory(ies) and move resources there to be managed with Terraform v0.12. This approach allows resources to be migrated incrementally and ensures existing resources can always be managed (e.g. emergency patches).

-Create a new Terraform [config directory](/architecture/concepts#organize) for *new* resources.
+Create a new Terraform [config directory](/architecture/concepts/#organize) for *new* resources.

 ```shell
 mkdir infra2
@ -404,7 +404,7 @@ tree .
 └── infraB  <- new Terraform v0.12.x configs
 ```

-Define Typhoon clusters in the new config directory using Terraform v0.12 syntax. Follow the Typhoon v1.17.0+ docs (e.g. use `terraform12` in the `infraB` dir). See [AWS](/cl/aws), [Azure](/cl/azure), [Bare-Metal](/cl/bare-metal), [Digital Ocean](/cl/digital-ocean), or [Google-Cloud](/cl/google-cloud)) to create new clusters. Follow the usual [upgrade](/topics/maintenance/#upgrades) process to apply workloads and shift traffic. Later, switch back to the old config directory and deprovision clusters with Terraform v0.11.
+Define Typhoon clusters in the new config directory using Terraform v0.12 syntax. Follow the Typhoon v1.15.0+ docs (e.g. use `terraform12` in the `infraB` dir). See [AWS](/cl/aws), [Azure](/cl/azure), [Bare-Metal](/cl/bare-metal), [Digital Ocean](/cl/digital-ocean), or [Google-Cloud](/cl/google-cloud)) to create new clusters. Follow the usual [upgrade](/topics/maintenance/#upgrades) process to apply workloads and shift traffic. Later, switch back to the old config directory and deprovision clusters with Terraform v0.11.

 ```shell
 terraform12 init
--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.17.0 (upstream)
+* Kubernetes v1.17.2 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/google-cloud/container-linux/kubernetes/bootstrap.tf
+++ b/google-cloud/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=24e5513ee6da4888005aac02101f73fc9e5e829f"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=05297b94a936c356851e180e4963034e0047e1c0"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/container-linux/kubernetes/cl/controller.yaml
+++ b/google-cloud/container-linux/kubernetes/cl/controller.yaml
@ -50,28 +50,46 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --insecure-options=image"
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -80,14 +98,15 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node.kubernetes.io/master \
          --node-labels=node.kubernetes.io/controller="true" \
          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --read-only-port=0 \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
        Restart=always
@ -113,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -128,14 +147,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/kubernetes/kubelet.env
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
    - path: /opt/bootstrap/layout
      filesystem: root
      mode: 0544
--- a/google-cloud/container-linux/kubernetes/controllers.tf
+++ b/google-cloud/container-linux/kubernetes/controllers.tf
@ -11,7 +11,7 @@ resource "google_dns_record_set" "etcds" {
  ttl  = 300

  # private IPv4 address for etcd
-  rrdatas = [element(google_compute_instance.controllers.*.network_interface.0.network_ip, count.index)]
+  rrdatas = [google_compute_instance.controllers.*.network_interface.0.network_ip[count.index]]
 }

 # Zones in the region
@ -29,12 +29,13 @@ locals {
 resource "google_compute_instance" "controllers" {
  count = var.controller_count

-  name         = "${var.cluster_name}-controller-${count.index}"
+  name = "${var.cluster_name}-controller-${count.index}"
+  # use a zone in the region and wrap around (e.g. controllers > zones)
  zone         = element(local.zones, count.index)
  machine_type = var.controller_type

  metadata = {
-    user-data = element(data.ct_config.controller-ignitions.*.rendered, count.index)
+    user-data = data.ct_config.controller-ignitions.*.rendered[count.index]
  }

  boot_disk {
@ -64,11 +65,8 @@ resource "google_compute_instance" "controllers" {

 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count = var.controller_count
-  content = element(
-    data.template_file.controller-configs.*.rendered,
-    count.index,
-  )
+  count        = var.controller_count
+  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
  snippets     = var.controller_clc_snippets
 }
@ -77,7 +75,7 @@ data "ct_config" "controller-ignitions" {
 data "template_file" "controller-configs" {
  count = var.controller_count

-  template = file("${path.module}/cl/controller.yaml.tmpl")
+  template = file("${path.module}/cl/controller.yaml")

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
--- a/google-cloud/container-linux/kubernetes/network.tf
+++ b/google-cloud/container-linux/kubernetes/network.tf
@ -126,6 +126,20 @@ resource "google_compute_firewall" "internal-node-exporter" {
  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
 }

+# Allow Prometheus to scrape kube-proxy metrics
+resource "google_compute_firewall" "internal-kube-proxy" {
+  name    = "${var.cluster_name}-internal-kube-proxy"
+  network = google_compute_network.network.name
+
+  allow {
+    protocol = "tcp"
+    ports    = [10249]
+  }
+
+  source_tags = ["${var.cluster_name}-worker"]
+  target_tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+}
+
 # Allow apiserver to access kubelets for exec, log, port-forward
 resource "google_compute_firewall" "internal-kubelet" {
  name    = "${var.cluster_name}-internal-kubelet"
--- a/google-cloud/container-linux/kubernetes/ssh.tf
+++ b/google-cloud/container-linux/kubernetes/ssh.tf
@ -2,7 +2,7 @@ locals {
  # format assets for distribution
  assets_bundle = [
    # header with the unpack location
-    for key, value in module.bootstrap.assets_dist:
+    for key, value in module.bootstrap.assets_dist :
    format("##### %s\n%s", key, value)
  ]
 }
@ -21,7 +21,7 @@ resource "null_resource" "copy-controller-secrets" {
    user    = "core"
    timeout = "15m"
  }
-  
+
  provisioner "file" {
    content     = join("\n", local.assets_bundle)
    destination = "$HOME/assets"
--- a/google-cloud/container-linux/kubernetes/versions.tf
+++ b/google-cloud/container-linux/kubernetes/versions.tf
@ -3,7 +3,7 @@
 terraform {
  required_version = "~> 0.12.6"
  required_providers {
-    google   = "~> 2.19"
+    google   = ">= 2.19, < 4.0"
    ct       = "~> 0.3"
    template = "~> 2.1"
    null     = "~> 2.1"
--- a/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -25,29 +25,46 @@ systemd:
        Description=Kubelet via Hyperkube
        Wants=rpc-statd.service
        [Service]
-        EnvironmentFile=/etc/kubernetes/kubelet.env
-        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --volume=resolv,kind=host,source=/etc/resolv.conf \
-          --mount volume=resolv,target=/etc/resolv.conf \
-          --volume var-lib-cni,kind=host,source=/var/lib/cni \
-          --mount volume=var-lib-cni,target=/var/lib/cni \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --hosts-entry=host \
-          --insecure-options=image"
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/cni
        ExecStartPre=/bin/mkdir -p /var/lib/calico
        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/lib/coreos/kubelet-wrapper \
+        ExecStart=/usr/bin/rkt run \
+          --uuid-file-save=/var/cache/kubelet-pod.uuid \
+          --stage1-from-dir=stage1-fly.aci \
+          --hosts-entry host \
+          --insecure-options=image \
+          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
+          --mount volume=etc-kubernetes,target=/etc/kubernetes \
+          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
+          --mount volume=etc-machine-id,target=/etc/machine-id \
+          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
+          --mount volume=etc-os-release,target=/etc/os-release \
+          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
+          --mount volume=etc-resolv,target=/etc/resolv.conf \
+          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
+          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
+          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
+          --mount volume=lib-modules,target=/lib/modules \
+          --volume run,kind=host,source=/run \
+          --mount volume=run,target=/run \
+          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
+          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
+          --volume var-lib-calico,kind=host,source=/var/lib/calico \
+          --mount volume=var-lib-calico,target=/var/lib/calico \
+          --volume var-lib-docker,kind=host,source=/var/lib/docker \
+          --mount volume=var-lib-docker,target=/var/lib/docker \
+          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
+          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
+          --volume var-log,kind=host,source=/var/log \
+          --mount volume=var-log,target=/var/log \
+          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
+          --mount volume=opt-cni-bin,target=/opt/cni/bin \
+          docker://k8s.gcr.io/hyperkube:v1.17.2 \
+          --exec=/usr/local/bin/kubelet -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -56,6 +73,7 @@ systemd:
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
+          --healthz-port=0 \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
@ -91,14 +109,6 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
-    - path: /etc/kubernetes/kubelet.env
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.17.0
-          KUBELET_IMAGE_ARGS="--exec=/usr/local/bin/kubelet"
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -116,7 +126,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.17.0 \
+            docker://k8s.gcr.io/hyperkube:v1.17.2 \
            --net=host \
            --dns=host \
            -- \
--- a/google-cloud/container-linux/kubernetes/workers/workers.tf
+++ b/google-cloud/container-linux/kubernetes/workers/workers.tf
@ -78,7 +78,7 @@ data "ct_config" "worker-ignition" {

 # Worker Container Linux config
 data "template_file" "worker-config" {
-  template = file("${path.module}/cl/worker.yaml.tmpl")
+  template = file("${path.module}/cl/worker.yaml")

  vars = {
    kubeconfig             = indent(10, var.kubeconfig)
--- a/requirements.txt
+++ b/requirements.txt
@ -1,4 +1,4 @@
 mkdocs==1.0.4
-mkdocs-material==4.5.1
+mkdocs-material==4.6.0
 pygments==2.4.2
 pymdown-extensions==6.2.0
Author	SHA1	Message	Date
Dalton Hubble	5643ad525f	Promote Fedora CoreOS from preview to alpha in docs * Add an announcement to the website as well	2020-01-23 08:47:18 -08:00
Dalton Hubble	d5b7ce8f27	Update kube-state-metrics from v1.9.2 to v1.9.3 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.3	2020-01-23 00:03:16 -08:00
Dalton Hubble	1cda5bcd2a	Update Kubernetes from v1.17.1 to v1.17.2 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.17.md#v1172	2020-01-21 18:27:39 -08:00
Dalton Hubble	bda73264f7	Update nginx-ingress from v0.26.1 to v0.27.1 * Change runAsUser from 33 to 101 for new alpine-based image * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.27.0 * https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.27.1	2020-01-20 15:22:16 -08:00
Dalton Hubble	dd930a2ff9	Update bare-metal Fedora CoreOS image location * Use Fedora CoreOS production download streams (change) * Use live PXE kernel and initramfs images * https://getfedora.org/coreos/download/ * Update docs example to use public images (cache is still recommended at large scale) and stable stream	2020-01-20 14:44:06 -08:00
Dalton Hubble	03ff3a9cf3	Update kube-state-metrics from v1.9.1 to v1.9.2 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.2	2020-01-18 15:32:10 -08:00
Dalton Hubble	48703f9906	Update Grafana from v6.5.2 to v6.5.3 * https://github.com/grafana/grafana/releases/tag/v6.5.3	2020-01-18 15:30:39 -08:00
Dalton Hubble	7ddd3d096d	Fix link in maintenance docs * Also a fix version mention, since Terraform v0.12 was added in Typhoon v1.15.0	2020-01-18 15:19:27 -08:00
Dalton Hubble	7daabd28b5	Update Calico from v3.11.1 to v3.11.2 * https://docs.projectcalico.org/v3.11/release-notes/	2020-01-18 13:45:24 -08:00
Dalton Hubble	b642e3b41b	Update Kubernetes from v1.17.0 to v1.17.1 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.17.md#v1171	2020-01-14 20:21:36 -08:00
Dalton Hubble	ac786a2efc	Update AWS Fedora CoreOS AMI filter for fedora-coreos-31 * Select the most recent fedora-coreos-31 AMI on AWS, instead of the most recent fedora-coreos-30 AMI (Nov 27, 2019) * Evaluated with fedora-coreos-31.20200108.2.0-hvm	2020-01-14 20:06:14 -08:00
Dalton Hubble	073fcb7067	Fix bare-metal instruction for watching install to disk * Original instructions were to watch install to disk by SSH'ing via port 2222 following Typhoon v1.10.1. Restore that message, since the version number in the instruction was incorrectly bumped on each release	2020-01-12 14:16:00 -08:00
Dalton Hubble	ce0569e03b	Remove unneeded Kubelet /var/run mount on Fedora CoreOS * /var/run symlinks to /run (already mounted)	2020-01-11 15:15:39 -08:00
Dalton Hubble	0e2fc89f78	Update kube-state-metrics from v1.9.0 to v1.9.1 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.1	2020-01-11 14:15:55 -08:00
Dalton Hubble	b1f521fc4a	Allow terraform-provider-google v3.x plugin versions * Typhoon Google Cloud is compatible with `terraform-provider-google` v3.x releases * No v3.x specific features are used, so v2.19+ provider versions are still allowed, to ease migrations	2020-01-11 14:07:18 -08:00
Dalton Hubble	73588cfad3	Update Prometheus from v2.15.1 to v2.15.2 * https://github.com/prometheus/prometheus/releases/tag/v2.15.2	2020-01-06 22:08:34 -08:00
Dalton Hubble	0223b31e1a	Ensure /etc/kubernetes exists following Kubelet inlining * Inlining the Kubelet service removed the need for the kubelet.env file declared in Ignition. However, on some platforms, this removed the guarantee that /etc/kubernetes exists. Bare-Metal and DigitalOcean distribute the kubelet kubeconfig through Terraform file provisioner (scp) and place it in (now missing) /etc/kubernetes * https://github.com/poseidon/typhoon/pull/606 * Fix bare-metal and DigitalOcean Ignition to ensure the desired directory exists following first boot from disk * Cloud platforms with worker pools distribute the kubeconfig through Ignition user data (no impact or need)	2020-01-06 21:38:20 -08:00
Dalton Hubble	bb586b60da	Reduce Prometheus addon's node-exporter tolerations * Change node-exporter DaemonSet tolerations from tolerating all possible NoSchedule taints to tolerating the master taint and the not ready taint (we'd like metrics regardless) * Users who add custom node taints must add their custom taints to the addon node-exporter DaemonSet. As an addon, its expected users copy and manipulate manifests out-of-band in their own systems	2020-01-06 21:24:24 -08:00
Dalton Hubble	43e05b9131	Enable kube-proxy metrics and allow Prometheus scrapes * Configure kube-proxy --metrics-bind-address=0.0.0.0 (default 127.0.0.1) to serve metrics on 0.0.0.0:10249 * Add firewall rules to allow Prometheus (resides on a worker) to scrape kube-proxy service endpoints on controllers or workers * Add a clusterIP: None service for kube-proxy endpoint discovery	2020-01-06 21:11:18 -08:00
Dalton Hubble	b2eb3e05d0	Disable Kubelet 127.0.0.1.10248 healthz endpoint * Kubelet runs a healthz server listening on 127.0.0.1:10248 by default. Its unused by Typhoon and can be disabled * https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/	2019-12-29 11:23:25 -08:00
Dalton Hubble	f1f4cd6fc0	Inline Container Linux kubelet.service, deprecate kubelet-wrapper * Change kubelet.service on Container Linux nodes to ExecStart Kubelet inline to replace the use of the host OS kubelet-wrapper script * Express rkt run flags and volume mounts in a clear, uniform way to make the Kubelet service easier to audit, manage, and understand * Eliminate reliance on a Container Linux kubelet-wrapper script * Typhoon for Fedora CoreOS developed a kubelet.service that similarly uses an inline ExecStart (except with podman instead of rkt) and a more minimal set of volume mounts. Adopt the volume improvements: * Change Kubelet /etc/kubernetes volume to read-only * Change Kubelet /etc/resolv.conf volume to read-only * Remove unneeded /var/lib/cni volume mount Background: * kubelet-wrapper was added in CoreOS around the time of Kubernetes v1.0 to simplify running a CoreOS-built hyperkube ACI image via rkt-fly. The script defaults are no longer ideal (e.g. rkt's notion of trust dates back to quay.io ACI image serving and signing, which informed the OCI standard images we use today, though they still lack rkt's signing ideas). * Shipping kubelet-wrapper was regretted at CoreOS, but remains in the distro for compatibility. The script is not updated to track hyperkube changes, but it is stable and kubelet.env overrides bridge most gaps * Typhoon Container Linux nodes have used kubelet-wrapper to rkt/rkt-fly run the Kubelet via the official k8s.gcr.io hyperkube image using overrides (new image registry, new image format, restart handling, new mounts, new entrypoint in v1.17). * Observation: Most of what it takes to run a Kubelet container is defined in Typhoon, not in kubelet-wrapper. The wrapper's value is now undermined by having to workaround its dated defaults. Typhoon may be better served defining Kubelet.service explicitly * Typhoon for Fedora CoreOS developed a kubelet.service without the use of a host OS kubelet-wrapper which is both clearer and eliminated some volume mounts	2019-12-29 11:17:26 -08:00
Dalton Hubble	50db3d0231	Rename CLC files and favor Terraform list index syntax * Rename Container Linux Config (CLC) files to .yaml to align with Fedora CoreOS Config (FCC) files and for syntax highlighting Replace common uses of Terraform `element` (which wraps around) with `list[index]` syntax to surface index errors	2019-12-28 12:14:01 -08:00
Dalton Hubble	11565ffa8a	Update Calico from v3.10.2 to v3.11.1 * https://docs.projectcalico.org/v3.11/release-notes/	2019-12-28 11:08:03 -08:00
Dalton Hubble	a4e843693f	Update Prometheus from v2.15.0 to v2.15.1 * https://github.com/prometheus/prometheus/releases/tag/v2.15.1	2019-12-26 09:12:55 -05:00
Dalton Hubble	f48e43c0b1	Update Prometheus from v2.14.0 to v2.15.0 * https://github.com/prometheus/prometheus/releases/tag/v2.15.0	2019-12-24 10:52:19 -05:00
Dalton Hubble	daa8d9d9ec	Update CoreDNS from v1.6.5 to v1.6.6 * https://coredns.io/2019/12/11/coredns-1.6.6-release/	2019-12-22 10:47:19 -05:00
Dalton Hubble	52d11096dc	Update kube-state-metrics from v1.9.0-rc.1 to v1.9.0 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.0 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.0-rc.1 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.0-rc.0	2019-12-20 13:53:37 -08:00
Dalton Hubble	00c431a9d2	Add Kubelet kubeconfig output for DigitalOcean * Allow the raw kubelet kubeconfig to be consumed via Terraform output	2019-12-18 23:20:55 -08:00
Dalton Hubble	0ecb995890	Update kube-state-metrics from v1.8.0 to v1.9.0-rc.1 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.0-rc.1 * https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.9.0-rc.0	2019-12-14 17:20:49 -08:00
Dalton Hubble	1b9fa2e688	Update Grafana from v6.5.1 to v6.5.2 * https://github.com/grafana/grafana/releases/tag/v6.5.2	2019-12-14 15:25:48 -08:00
Dalton Hubble	2d8e367664	Update mkdocs-material from v4.5.1 to v4.6.0	2019-12-14 15:02:28 -08:00