Update CHANGES for v1.18.1 release

* Change order of modules in the README
Add support for Fedora CoreOS on DigitalOcean
2025-08-03 02:01:33 +02:00 · 2020-04-11 13:23:49 -07:00 · 2020-04-09 23:55:29 -07:00 · 2020-04-08 19:41:48 -07:00 · 2020-04-06 21:09:25 -07:00 · 2020-04-04 13:13:19 -07:00
90 changed files with 1757 additions and 560 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,37 @@ Notable changes between versions.

 ## Latest

+## v1.18.1
+
+* Kubernetes [v1.18.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1181)
+* Choose Fedora CoreOS or Flatcar Linux (**action recommended**)
+  * Use a `fedora-coreos` module for Fedora CoreOS
+  * Use a `container-linux` module with OS set to Flatcar Linux
+* Update etcd from v3.4.5 to [v3.4.7](https://github.com/etcd-io/etcd/releases/tag/v3.4.7)
+* Change `kube-proxy` and `calico` or `flannel` to tolerate specific taints ([#682](https://github.com/poseidon/typhoon/pull/682))
+  * Tolerate master and not-ready taints, rather than tolerating all taints
+* Update flannel from v0.11.0 to v0.12.0 ([#690](https://github.com/poseidon/typhoon/pull/690))
+* Fix bootstrap when `networking` mode `flannel` (non-default) is chosen ([#689](https://github.com/poseidon/typhoon/pull/689))
+  * Regressed in v1.18.0 changes for Calico ([#675](https://github.com/poseidon/typhoon/pull/675))
+* Rename Container Linux `controller_clc_snippets` to `controller_snippets` for consistency ([#688](https://github.com/poseidon/typhoon/pull/688))
+* Rename Container Linux `worker_clc_snippets` to `worker_snippets` for consistency
+* Rename Container Linux `clc_snippets` (bare-metal) to `snippets` for consistency
+* Drop support for [gitRepo](https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo) volumes
+
+#### Azure
+
+* Fix Azure worker UDP outbound connections ([#691](https://github.com/poseidon/typhoon/pull/691))
+  * Fix Azure worker clock sync timeouts
+
+#### DigitalOcean
+
+* Add support for Fedora CoreOS ([#699](https://github.com/poseidon/typhoon/pull/699))
+
+#### Addons
+
+* Refresh Prometheus rules/alerts and Grafana dashboards ([#692](https://github.com/poseidon/typhoon/pull/692))
+* Update Grafana from v6.7.1 to v6.7.2
+
 ## v1.18.0

 * Kubernetes [v1.18.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1180)
@ -31,7 +62,7 @@ Notable changes between versions.

 #### Bare-Metal

-* Update default `os_stream` from testing to stable
+* Update Fedora CoreOS default `os_stream` from testing to stable

 #### Google Cloud

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@ -21,20 +21,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 Typhoon provides a Terraform Module for each supported operating system and platform.

-| Platform      | Operating System | Terraform Module | Status |
-|---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| Azure         | Container Linux  | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |
-
 Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | stable |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
+| DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | alpha |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |

 Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org/releases/).
@ -44,14 +37,24 @@ Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org
 | AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
 | Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | alpha |
 | Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | alpha |
-| Digital Ocean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | alpha |
+
+Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com/os/eol/) after May 2020).
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
+| Azure         | Container Linux  | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
+| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
+| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |

 ## Documentation

 * [Docs](https://typhoon.psdn.io)
 * Architecture [concepts](https://typhoon.psdn.io/architecture/concepts/) and [operating systems](https://typhoon.psdn.io/architecture/operating-systems/)
-* Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
+* Fedora CoreOS tutorials for [AWS](docs/fedora-coreos/aws.md), [Bare-Metal](docs/fedora-coreos/bare-metal.md), [DigitalOcean](docs/fedora-coreos/digitalocean.md), and [Google Cloud](docs/fedora-coreos/google-cloud.md)
 * Flatcar Linux tutorials for [AWS](docs/cl/aws.md), [Azure](docs/cl/azure.md), [Bare-Metal](docs/cl/bare-metal.md), [DigitalOcean](docs/cl/digital-ocean.md), and [Google Cloud](docs/cl/google-cloud.md)

 ## Usage
@ -60,7 +63,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -99,9 +102,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
 ```

 List the pods.
--- a/addons/grafana/dashboards-k8s-resources-1.yaml
+++ b/addons/grafana/dashboards-k8s-resources-1.yaml
@ -59,7 +59,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\", cluster=\"$cluster\"}[1m]))",
+                  "expr": "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\", cluster=\"$cluster\"}[$__interval]))",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -1561,7 +1561,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1570,7 +1570,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1579,7 +1579,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1588,7 +1588,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1597,7 +1597,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1606,7 +1606,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1706,7 +1706,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -1804,7 +1804,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -1902,7 +1902,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2000,7 +2000,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2098,7 +2098,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2196,7 +2196,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2294,7 +2294,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2392,7 +2392,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2499,41 +2499,6 @@ data:
            "type": "query",
            "useTags": false
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "5m",
-              "value": "5m"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "interval",
-            "options": [
-              {
-                "selected": true,
-                "text": "4h",
-                "value": "4h"
-              }
-            ],
-            "query": "4h",
-            "refresh": 2,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "interval",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -4033,7 +3998,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4042,7 +4007,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4051,7 +4016,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4060,7 +4025,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4069,7 +4034,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4078,7 +4043,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4178,7 +4143,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4276,7 +4241,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4374,7 +4339,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4472,7 +4437,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4570,7 +4535,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4668,7 +4633,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4748,41 +4713,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "5m",
-              "value": "5m"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "interval",
-            "options": [
-              {
-                "selected": true,
-                "text": "4h",
-                "value": "4h"
-              }
-            ],
-            "query": "4h",
-            "refresh": 2,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "interval",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -5724,41 +5654,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "5m",
-              "value": "5m"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "interval",
-            "options": [
-              {
-                "selected": true,
-                "text": "4h",
-                "value": "4h"
-              }
-            ],
-            "query": "4h",
-            "refresh": 2,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "interval",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
--- a/addons/grafana/dashboards-k8s-resources-2.yaml
+++ b/addons/grafana/dashboards-k8s-resources-2.yaml
@ -1042,7 +1042,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1140,7 +1140,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1238,7 +1238,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1336,7 +1336,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1434,7 +1434,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1532,7 +1532,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1612,41 +1612,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "5m",
-              "value": "5m"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "interval",
-            "options": [
-              {
-                "selected": true,
-                "text": "4h",
-                "value": "4h"
-              }
-            ],
-            "query": "4h",
-            "refresh": 2,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "interval",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -2701,7 +2666,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2710,7 +2675,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2719,7 +2684,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2728,7 +2693,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2737,7 +2702,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2746,7 +2711,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2846,7 +2811,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -2944,7 +2909,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3042,7 +3007,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3140,7 +3105,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3238,7 +3203,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3336,7 +3301,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3434,7 +3399,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3532,7 +3497,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod) \ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3612,41 +3577,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "5m",
-              "value": "5m"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "interval",
-            "options": [
-              {
-                "selected": true,
-                "text": "4h",
-                "value": "4h"
-              }
-            ],
-            "query": "4h",
-            "refresh": 2,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "interval",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -3708,7 +3638,7 @@ data:
              "value": ""
            },
            "datasource": "$datasource",
-            "hide": 2,
+            "hide": 0,
            "includeAll": false,
            "label": null,
            "multi": false,
@ -4906,7 +4836,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4915,7 +4845,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4924,7 +4854,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4933,7 +4863,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4942,7 +4872,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4951,7 +4881,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5051,7 +4981,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5149,7 +5079,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5247,7 +5177,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5345,7 +5275,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5443,7 +5373,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5541,7 +5471,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5639,7 +5569,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5737,7 +5667,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$interval])\n* on (namespace,pod) \ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod) \ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5817,41 +5747,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "5m",
-              "value": "5m"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "interval",
-            "options": [
-              {
-                "selected": true,
-                "text": "4h",
-                "value": "4h"
-              }
-            ],
-            "query": "4h",
-            "refresh": 2,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "interval",
-            "useTags": false
-          },
          {
            "allValue": null,
            "auto": false,
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -23,7 +23,7 @@ spec:
    spec:
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:6.7.1
+          image: docker.io/grafana/grafana:6.7.2
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -252,25 +252,25 @@ data:
          "name": "kube-apiserver.rules",
          "rules": [
            {
-              "expr": "sum(rate(apiserver_request_duration_seconds_sum{subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|PROXY|CONNECT\"}[5m])) without(instance, pod)\n/\nsum(rate(apiserver_request_duration_seconds_count{subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|PROXY|CONNECT\"}[5m])) without(instance, pod)\n",
+              "expr": "sum(rate(apiserver_request_duration_seconds_sum{subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod)\n/\nsum(rate(apiserver_request_duration_seconds_count{subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod)\n",
              "record": "cluster:apiserver_request_duration_seconds:mean5m"
            },
            {
-              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
+              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
              "labels": {
                "quantile": "0.99"
              },
              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
            },
            {
-              "expr": "histogram_quantile(0.9, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
+              "expr": "histogram_quantile(0.9, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
              "labels": {
                "quantile": "0.9"
              },
              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
            },
            {
-              "expr": "histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
+              "expr": "histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
              "labels": {
                "quantile": "0.5"
              },
@ -805,6 +805,7 @@ data:
            {
              "alert": "ErrorBudgetBurn",
              "annotations": {
+                "message": "High requests error budget burn for job=apiserver (current value: {{ $value }})",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-errorbudgetburn"
              },
              "expr": "(\n  status_class_5xx:apiserver_request_total:ratio_rate1h{job=\"apiserver\"} > (14.4*0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate5m{job=\"apiserver\"} > (14.4*0.010000)\n)\nor\n(\n  status_class_5xx:apiserver_request_total:ratio_rate6h{job=\"apiserver\"} > (6*0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate30m{job=\"apiserver\"} > (6*0.010000)\n)\n",
@ -816,6 +817,7 @@ data:
            {
              "alert": "ErrorBudgetBurn",
              "annotations": {
+                "message": "High requests error budget burn for job=apiserver (current value: {{ $value }})",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-errorbudgetburn"
              },
              "expr": "(\n  status_class_5xx:apiserver_request_total:ratio_rate1d{job=\"apiserver\"} > (3*0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate2h{job=\"apiserver\"} > (3*0.010000)\n)\nor\n(\n  status_class_5xx:apiserver_request_total:ratio_rate3d{job=\"apiserver\"} > (0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate6h{job=\"apiserver\"} > (0.010000)\n)\n",
@ -853,30 +855,6 @@ data:
                "severity": "critical"
              }
            },
-            {
-              "alert": "KubeAPIErrorsHigh",
-              "annotations": {
-                "message": "API server is returning errors for {{ $value | humanizePercentage }} of requests.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
-              },
-              "expr": "sum(rate(apiserver_request_total{job=\"apiserver\",code=~\"5..\"}[5m]))\n  /\nsum(rate(apiserver_request_total{job=\"apiserver\"}[5m])) > 0.03\n",
-              "for": "10m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
-            {
-              "alert": "KubeAPIErrorsHigh",
-              "annotations": {
-                "message": "API server is returning errors for {{ $value | humanizePercentage }} of requests.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
-              },
-              "expr": "sum(rate(apiserver_request_total{job=\"apiserver\",code=~\"5..\"}[5m]))\n  /\nsum(rate(apiserver_request_total{job=\"apiserver\"}[5m])) > 0.01\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
            {
              "alert": "KubeAPIErrorsHigh",
              "annotations": {
@ -993,7 +971,7 @@ data:
                "message": "Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods"
              },
-              "expr": "max(max(kubelet_running_pod_count{job=\"kubelet\"}) by(instance) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"}) by(node) / max(kube_node_status_capacity_pods{job=\"kube-state-metrics\"}) by(node) > 0.95\n",
+              "expr": "max(max(kubelet_running_pod_count{job=\"kubelet\"}) by(instance) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"}) by(node) / max(kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) by(node) > 0.95\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -1029,7 +1007,7 @@ data:
                "message": "Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }}.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh"
              },
-              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\"}[5m])) by (instance, le)) * on(instance) group_left(node) kubelet_node_name  > 5\n",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\"}[5m])) by (instance, le)) * on(instance) group_left(node) kubelet_node_name  > 60\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -1085,9 +1063,167 @@ data:
        }
      ]
    }
+  loki.yaml: |-
+    {
+      "groups": [
+        {
+          "name": "loki_rules",
+          "rules": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job))",
+              "record": "job:loki_request_duration_seconds:99quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job))",
+              "record": "job:loki_request_duration_seconds:50quantile"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job) / sum(rate(loki_request_duration_seconds_count[1m])) by (job)",
+              "record": "job:loki_request_duration_seconds:avg"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job)",
+              "record": "job:loki_request_duration_seconds_bucket:sum_rate"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job)",
+              "record": "job:loki_request_duration_seconds_sum:sum_rate"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_count[1m])) by (job)",
+              "record": "job:loki_request_duration_seconds_count:sum_rate"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route))",
+              "record": "job_route:loki_request_duration_seconds:99quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route))",
+              "record": "job_route:loki_request_duration_seconds:50quantile"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job, route) / sum(rate(loki_request_duration_seconds_count[1m])) by (job, route)",
+              "record": "job_route:loki_request_duration_seconds:avg"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route)",
+              "record": "job_route:loki_request_duration_seconds_bucket:sum_rate"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job, route)",
+              "record": "job_route:loki_request_duration_seconds_sum:sum_rate"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_count[1m])) by (job, route)",
+              "record": "job_route:loki_request_duration_seconds_count:sum_rate"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, namespace, job, route))",
+              "record": "namespace_job_route:loki_request_duration_seconds:99quantile"
+            },
+            {
+              "expr": "histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, namespace, job, route))",
+              "record": "namespace_job_route:loki_request_duration_seconds:50quantile"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (namespace, job, route) / sum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route)",
+              "record": "namespace_job_route:loki_request_duration_seconds:avg"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, namespace, job, route)",
+              "record": "namespace_job_route:loki_request_duration_seconds_bucket:sum_rate"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (namespace, job, route)",
+              "record": "namespace_job_route:loki_request_duration_seconds_sum:sum_rate"
+            },
+            {
+              "expr": "sum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route)",
+              "record": "namespace_job_route:loki_request_duration_seconds_count:sum_rate"
+            }
+          ]
+        },
+        {
+          "name": "loki_alerts",
+          "rules": [
+            {
+              "alert": "LokiRequestErrors",
+              "annotations": {
+                "message": "{{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}% errors.\n"
+              },
+              "expr": "100 * sum(rate(loki_request_duration_seconds_count{status_code=~\"5..\"}[1m])) by (namespace, job, route)\n  /\nsum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route)\n  > 10\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "LokiRequestLatency",
+              "annotations": {
+                "message": "{{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency.\n"
+              },
+              "expr": "namespace_job_route:loki_request_duration_seconds:99quantile{route!~\"(?i).*tail.*\"} > 1\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            }
+          ]
+        }
+      ]
+    }
  node-exporter.yaml: |-
    {
      "groups": [
+        {
+          "name": "node-exporter.rules",
+          "rules": [
+            {
+              "expr": "count without (cpu) (\n  count without (mode) (\n    node_cpu_seconds_total{job=\"node-exporter\"}\n  )\n)\n",
+              "record": "instance:node_num_cpu:sum"
+            },
+            {
+              "expr": "1 - avg without (cpu, mode) (\n  rate(node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\"}[1m])\n)\n",
+              "record": "instance:node_cpu_utilisation:rate1m"
+            },
+            {
+              "expr": "(\n  node_load1{job=\"node-exporter\"}\n/\n  instance:node_num_cpu:sum{job=\"node-exporter\"}\n)\n",
+              "record": "instance:node_load1_per_cpu:ratio"
+            },
+            {
+              "expr": "1 - (\n  node_memory_MemAvailable_bytes{job=\"node-exporter\"}\n/\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n)\n",
+              "record": "instance:node_memory_utilisation:ratio"
+            },
+            {
+              "expr": "rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[1m])\n",
+              "record": "instance:node_vmstat_pgmajfault:rate1m"
+            },
+            {
+              "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[1m])\n",
+              "record": "instance_device:node_disk_io_time_seconds:rate1m"
+            },
+            {
+              "expr": "rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[1m])\n",
+              "record": "instance_device:node_disk_io_time_weighted_seconds:rate1m"
+            },
+            {
+              "expr": "sum without (device) (\n  rate(node_network_receive_bytes_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "record": "instance:node_network_receive_bytes_excluding_lo:rate1m"
+            },
+            {
+              "expr": "sum without (device) (\n  rate(node_network_transmit_bytes_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "record": "instance:node_network_transmit_bytes_excluding_lo:rate1m"
+            },
+            {
+              "expr": "sum without (device) (\n  rate(node_network_receive_drop_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "record": "instance:node_network_receive_drop_excluding_lo:rate1m"
+            },
+            {
+              "expr": "sum without (device) (\n  rate(node_network_transmit_drop_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
+              "record": "instance:node_network_transmit_drop_excluding_lo:rate1m"
+            }
+          ]
+        },
        {
          "name": "node-exporter",
          "rules": [
@ -1210,6 +1346,41 @@ data:
              "labels": {
                "severity": "warning"
              }
+            },
+            {
+              "alert": "NodeHighNumberConntrackEntriesUsed",
+              "annotations": {
+                "description": "{{ $value | humanizePercentage }} of conntrack entries are used",
+                "summary": "Number of conntrack are getting close to the limit"
+              },
+              "expr": "(node_nf_conntrack_entries / node_nf_conntrack_entries_limit) > 0.75\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeClockSkewDetected",
+              "annotations": {
+                "message": "Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.",
+                "summary": "Clock skew detected."
+              },
+              "expr": "(\n  node_timex_offset_seconds > 0.05\nand\n  deriv(node_timex_offset_seconds[5m]) >= 0\n)\nor\n(\n  node_timex_offset_seconds < -0.05\nand\n  deriv(node_timex_offset_seconds[5m]) <= 0\n)\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeClockNotSynchronising",
+              "annotations": {
+                "message": "Clock on {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.",
+                "summary": "Clock not synchronising."
+              },
+              "expr": "min_over_time(node_timex_sync_status[5m]) == 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
            }
          ]
        }
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/container-linux/kubernetes/cl/controller.yaml
+++ b/aws/container-linux/kubernetes/cl/controller.yaml
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.5"
+            Environment="ETCD_IMAGE_TAG=v3.4.7"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -91,7 +91,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -134,7 +134,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -170,9 +170,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
      mode: 0544
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -52,7 +52,7 @@ data "ct_config" "controller-ignitions" {
  count        = var.controller_count
  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
-  snippets     = var.controller_clc_snippets
+  snippets     = var.controller_snippets
 }

 # Controller Container Linux configs
--- a/aws/container-linux/kubernetes/network.tf
+++ b/aws/container-linux/kubernetes/network.tf
@ -31,15 +31,15 @@ resource "aws_route_table" "default" {
 }

 resource "aws_route" "egress-ipv4" {
-  route_table_id = aws_route_table.default.id
+  route_table_id         = aws_route_table.default.id
  destination_cidr_block = "0.0.0.0/0"
-  gateway_id = aws_internet_gateway.gateway.id
+  gateway_id             = aws_internet_gateway.gateway.id
 }

 resource "aws_route" "egress-ipv6" {
-  route_table_id = aws_route_table.default.id
+  route_table_id              = aws_route_table.default.id
  destination_ipv6_cidr_block = "::/0"
-  gateway_id = aws_internet_gateway.gateway.id
+  gateway_id                  = aws_internet_gateway.gateway.id
 }

 # Subnets (one per availability zone)
--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -77,13 +77,13 @@ variable "worker_target_groups" {
  default     = []
 }

-variable "controller_clc_snippets" {
+variable "controller_snippets" {
  type        = list(string)
  description = "Controller Container Linux Config snippets"
  default     = []
 }

-variable "worker_clc_snippets" {
+variable "worker_snippets" {
  type        = list(string)
  description = "Worker Container Linux Config snippets"
  default     = []
--- a/aws/container-linux/kubernetes/workers.tf
+++ b/aws/container-linux/kubernetes/workers.tf
@ -18,7 +18,7 @@ module "workers" {
  ssh_authorized_key    = var.ssh_authorized_key
  service_cidr          = var.service_cidr
  cluster_domain_suffix = var.cluster_domain_suffix
-  clc_snippets          = var.worker_clc_snippets
+  snippets              = var.worker_snippets
  node_labels           = var.worker_node_labels
 }

--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml
@ -64,7 +64,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -127,7 +127,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -70,7 +70,7 @@ variable "target_groups" {
  default     = []
 }

-variable "clc_snippets" {
+variable "snippets" {
  type        = list(string)
  description = "Container Linux Config snippets"
  default     = []
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -73,7 +73,7 @@ resource "aws_launch_configuration" "worker" {
 data "ct_config" "worker-ignition" {
  content      = data.template_file.worker-config.rendered
  pretty_print = false
-  snippets     = var.clc_snippets
+  snippets     = var.snippets
 }

 # Worker Container Linux config
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/aws/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.5
+          quay.io/coreos/etcd:v3.4.7
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -79,7 +79,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.0 \
+          quay.io/poseidon/kubelet:v1.18.1 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -122,7 +122,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.0
+            quay.io/poseidon/kubelet:v1.18.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -155,9 +155,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
--- a/aws/fedora-coreos/kubernetes/network.tf
+++ b/aws/fedora-coreos/kubernetes/network.tf
@ -31,15 +31,15 @@ resource "aws_route_table" "default" {
 }

 resource "aws_route" "egress-ipv4" {
-  route_table_id = aws_route_table.default.id
+  route_table_id         = aws_route_table.default.id
  destination_cidr_block = "0.0.0.0/0"
-  gateway_id = aws_internet_gateway.gateway.id
+  gateway_id             = aws_internet_gateway.gateway.id
 }

 resource "aws_route" "egress-ipv6" {
-  route_table_id = aws_route_table.default.id
+  route_table_id              = aws_route_table.default.id
  destination_ipv6_cidr_block = "::/0"
-  gateway_id = aws_internet_gateway.gateway.id
+  gateway_id                  = aws_internet_gateway.gateway.id
 }

 # Subnets (one per availability zone)
--- a/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -49,7 +49,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.0 \
+          quay.io/poseidon/kubelet:v1.18.1 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -87,7 +87,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.0 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/azure/container-linux/kubernetes/bootstrap.tf
+++ b/azure/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/azure/container-linux/kubernetes/cl/controller.yaml
+++ b/azure/container-linux/kubernetes/cl/controller.yaml
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.5"
+            Environment="ETCD_IMAGE_TAG=v3.4.7"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -90,7 +90,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -168,9 +168,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
      mode: 0544
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -65,9 +65,9 @@ resource "azurerm_linux_virtual_machine" "controllers" {
    for_each = local.flavor == "flatcar" ? [1] : []

    content {
-      name = local.channel
+      name      = local.channel
      publisher = "kinvolk"
-      product = "flatcar-container-linux"
+      product   = "flatcar-container-linux"
    }
  }

@ -141,7 +141,7 @@ data "ct_config" "controller-ignitions" {
  count        = var.controller_count
  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
-  snippets     = var.controller_clc_snippets
+  snippets     = var.controller_snippets
 }

 # Controller Container Linux configs
--- a/azure/container-linux/kubernetes/lb.tf
+++ b/azure/container-linux/kubernetes/lb.tf
@ -72,6 +72,7 @@ resource "azurerm_lb_rule" "ingress-http" {
  name                           = "ingress-http"
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "ingress"
+  disable_outbound_snat          = true

  protocol                = "Tcp"
  frontend_port           = 80
@ -86,6 +87,7 @@ resource "azurerm_lb_rule" "ingress-https" {
  name                           = "ingress-https"
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "ingress"
+  disable_outbound_snat          = true

  protocol                = "Tcp"
  frontend_port           = 443
@ -94,6 +96,20 @@ resource "azurerm_lb_rule" "ingress-https" {
  probe_id                = azurerm_lb_probe.ingress.id
 }

+# Worker outbound TCP/UDP SNAT
+resource "azurerm_lb_outbound_rule" "worker-outbound" {
+  resource_group_name = azurerm_resource_group.cluster.name
+
+  name            = "worker"
+  loadbalancer_id = azurerm_lb.cluster.id
+  frontend_ip_configuration {
+    name = "ingress"
+  }
+
+  protocol                = "All"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+}
+
 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
  resource_group_name = azurerm_resource_group.cluster.name
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -64,13 +64,13 @@ variable "worker_priority" {
  default     = "Regular"
 }

-variable "controller_clc_snippets" {
+variable "controller_snippets" {
  type        = list(string)
  description = "Controller Container Linux Config snippets"
  default     = []
 }

-variable "worker_clc_snippets" {
+variable "worker_snippets" {
  type        = list(string)
  description = "Worker Container Linux Config snippets"
  default     = []
--- a/azure/container-linux/kubernetes/workers.tf
+++ b/azure/container-linux/kubernetes/workers.tf
@ -19,6 +19,6 @@ module "workers" {
  ssh_authorized_key    = var.ssh_authorized_key
  service_cidr          = var.service_cidr
  cluster_domain_suffix = var.cluster_domain_suffix
-  clc_snippets          = var.worker_clc_snippets
+  snippets              = var.worker_snippets
  node_labels           = var.worker_node_labels
 }
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml
@ -63,7 +63,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -125,7 +125,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@ -56,7 +56,7 @@ variable "priority" {
  default     = "Regular"
 }

-variable "clc_snippets" {
+variable "snippets" {
  type        = list(string)
  description = "Container Linux Config snippets"
  default     = []
--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -9,19 +9,19 @@ locals {
 resource "azurerm_linux_virtual_machine_scale_set" "workers" {
  resource_group_name = var.resource_group_name

-  name                   = "${var.name}-worker"
-  location               = var.region
-  sku = var.vm_type
+  name      = "${var.name}-worker"
+  location  = var.region
+  sku       = var.vm_type
  instances = var.worker_count
  # instance name prefix for instances in the set
-  computer_name_prefix = "${var.name}-worker"
+  computer_name_prefix   = "${var.name}-worker"
  single_placement_group = false
-  custom_data          = base64encode(data.ct_config.worker-ignition.rendered)
+  custom_data            = base64encode(data.ct_config.worker-ignition.rendered)

  # storage
  os_disk {
    storage_account_type = "Standard_LRS"
-    caching           = "ReadWrite"
+    caching              = "ReadWrite"
  }

  source_image_reference {
@ -36,16 +36,16 @@ resource "azurerm_linux_virtual_machine_scale_set" "workers" {
    for_each = local.flavor == "flatcar" ? [1] : []

    content {
-      name = local.channel
+      name      = local.channel
      publisher = "kinvolk"
-      product = "flatcar-container-linux"
+      product   = "flatcar-container-linux"
    }
  }

  # Azure requires setting admin_ssh_key, though Ignition custom_data handles it too
  admin_username = "core"
  admin_ssh_key {
-    username = "core"
+    username   = "core"
    public_key = var.ssh_authorized_key
  }

@ -98,7 +98,7 @@ resource "azurerm_monitor_autoscale_setting" "workers" {
 data "ct_config" "worker-ignition" {
  content      = data.template_file.worker-config.rendered
  pretty_print = false
-  snippets     = var.clc_snippets
+  snippets     = var.snippets
 }

 # Worker Container Linux configs
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/container-linux/kubernetes/bootstrap.tf
+++ b/bare-metal/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.5"
+            Environment="ETCD_IMAGE_TAG=v3.4.7"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -103,7 +103,7 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -147,7 +147,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -186,9 +186,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
      mode: 0544
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml
@ -76,7 +76,7 @@ systemd:
          --mount volume=etc-iscsi,target=/etc/iscsi \
          --volume usr-sbin-iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=usr-sbin-iscsiadm,target=/sbin/iscsiadm \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -144,7 +144,7 @@ data "ct_config" "controller-ignitions" {
  count        = length(var.controllers)
  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
-  snippets     = local.clc_map[var.controllers.*.name[count.index]]
+  snippets     = lookup(var.snippets, var.controllers.*.name[count.index], [])
 }

 data "template_file" "controller-configs" {
@ -174,7 +174,7 @@ data "ct_config" "worker-ignitions" {
  count        = length(var.workers)
  content      = data.template_file.worker-configs.*.rendered[count.index]
  pretty_print = false
-  snippets     = local.clc_map[var.workers.*.name[count.index]]
+  snippets     = lookup(var.snippets, var.workers.*.name[count.index], [])
 }

 data "template_file" "worker-configs" {
@ -192,24 +192,3 @@ data "template_file" "worker-configs" {
    node_taints            = join(",", lookup(var.worker_node_taints, var.workers.*.name[count.index], []))
  }
 }
-
-locals {
-  # Hack to workaround https://github.com/hashicorp/terraform/issues/17251
-  # Still an issue in Terraform v0.12 https://github.com/hashicorp/terraform/issues/20572
-  # Default Container Linux config snippets map every node names to list("\n") so
-  # all lookups succeed
-  clc_defaults = zipmap(
-    concat(var.controllers.*.name, var.workers.*.name),
-    chunklist(data.template_file.clc-default-snippets.*.rendered, 1),
-  )
-
-  # Union of the default and user specific snippets, later overrides prior.
-  clc_map = merge(local.clc_defaults, var.clc_snippets)
-}
-
-// Horrible hack to generate a Terraform list of node count length
-data "template_file" "clc-default-snippets" {
-  count    = length(var.controllers) + length(var.workers)
-  template = "\n"
-}
-
--- a/bare-metal/container-linux/kubernetes/variables.tf
+++ b/bare-metal/container-linux/kubernetes/variables.tf
@ -49,22 +49,22 @@ List of worker machine details (unique name, identifying MAC address, FQDN)
 EOD
 }

-variable "clc_snippets" {
+variable "snippets" {
  type        = map(list(string))
  description = "Map from machine names to lists of Container Linux Config snippets"
  default     = {}
 }

 variable "worker_node_labels" {
-  type = map(list(string))
+  type        = map(list(string))
  description = "Map from worker names to lists of initial node labels"
-  default = {}
+  default     = {}
 }

 variable "worker_node_taints" {
-  type = map(list(string))
+  type        = map(list(string))
  description = "Map from worker names to lists of initial node taints"
-  default = {}
+  default     = {}
 }

 # configuration
--- a/bare-metal/fedora-coreos/kubernetes/README.md
+++ b/bare-metal/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
+++ b/bare-metal/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name                    = var.cluster_name
  api_servers                     = [var.k8s_domain_name]
--- a/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.5
+          quay.io/coreos/etcd:v3.4.7
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -80,7 +80,7 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          quay.io/poseidon/kubelet:v1.18.0 \
+          quay.io/poseidon/kubelet:v1.18.1 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -133,7 +133,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.0
+            quay.io/poseidon/kubelet:v1.18.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -166,9 +166,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
--- a/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/bare-metal/fedora-coreos/kubernetes/fcc/worker.yaml
@ -50,7 +50,7 @@ systemd:
          --volume /opt/cni/bin:/opt/cni/bin:z \
          --volume /etc/iscsi:/etc/iscsi \
          --volume /sbin/iscsiadm:/sbin/iscsiadm \
-          quay.io/poseidon/kubelet:v1.18.0 \
+          quay.io/poseidon/kubelet:v1.18.1 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
--- a/bare-metal/fedora-coreos/kubernetes/profiles.tf
+++ b/bare-metal/fedora-coreos/kubernetes/profiles.tf
@ -48,8 +48,8 @@ resource "matchbox_profile" "controllers" {
 data "ct_config" "controller-ignitions" {
  count = length(var.controllers)

-  content = data.template_file.controller-configs.*.rendered[count.index]
-  strict  = true
+  content  = data.template_file.controller-configs.*.rendered[count.index]
+  strict   = true
  snippets = lookup(var.snippets, var.controllers.*.name[count.index], [])
 }

@ -84,8 +84,8 @@ resource "matchbox_profile" "workers" {
 data "ct_config" "worker-ignitions" {
  count = length(var.workers)

-  content = data.template_file.worker-configs.*.rendered[count.index]
-  strict  = true
+  content  = data.template_file.worker-configs.*.rendered[count.index]
+  strict   = true
  snippets = lookup(var.snippets, var.workers.*.name[count.index], [])
 }

--- a/bare-metal/fedora-coreos/kubernetes/variables.tf
+++ b/bare-metal/fedora-coreos/kubernetes/variables.tf
@ -57,15 +57,15 @@ variable "snippets" {
 }

 variable "worker_node_labels" {
-  type = map(list(string))
+  type        = map(list(string))
  description = "Map from worker names to lists of initial node labels"
-  default = {}
+  default     = {}
 }

 variable "worker_node_taints" {
-  type = map(list(string))
+  type        = map(list(string))
  description = "Map from worker names to lists of initial node taints"
-  default = {}
+  default     = {}
 }

 # configuration
--- a/digital-ocean/container-linux/kubernetes/README.md
+++ b/digital-ocean/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/digital-ocean/container-linux/kubernetes/bootstrap.tf
+++ b/digital-ocean/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/digital-ocean/container-linux/kubernetes/cl/controller.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/controller.yaml
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.5"
+            Environment="ETCD_IMAGE_TAG=v3.4.7"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -101,7 +101,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -144,7 +144,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -177,9 +177,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
      mode: 0544
--- a/digital-ocean/container-linux/kubernetes/cl/worker.yaml
+++ b/digital-ocean/container-linux/kubernetes/cl/worker.yaml
@ -74,7 +74,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -131,7 +131,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/digital-ocean/container-linux/kubernetes/controllers.tf
+++ b/digital-ocean/container-linux/kubernetes/controllers.tf
@ -1,5 +1,5 @@
 locals {
-  official_images = ["coreos-stable", "coreos-beta", "coreos-alpha"]
+  official_images   = ["coreos-stable", "coreos-beta", "coreos-alpha"]
  is_official_image = contains(local.official_images, var.os_image)
 }

@ -72,7 +72,7 @@ data "ct_config" "controller-ignitions" {
  count        = var.controller_count
  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
-  snippets     = var.controller_clc_snippets
+  snippets     = var.controller_snippets
 }

 # Controller Container Linux configs
--- a/digital-ocean/container-linux/kubernetes/variables.tf
+++ b/digital-ocean/container-linux/kubernetes/variables.tf
@ -47,13 +47,13 @@ variable "os_image" {
  default     = "coreos-stable"
 }

-variable "controller_clc_snippets" {
+variable "controller_snippets" {
  type        = list(string)
  description = "Controller Container Linux Config snippets"
  default     = []
 }

-variable "worker_clc_snippets" {
+variable "worker_snippets" {
  type        = list(string)
  description = "Worker Container Linux Config snippets"
  default     = []
--- a/digital-ocean/container-linux/kubernetes/workers.tf
+++ b/digital-ocean/container-linux/kubernetes/workers.tf
@ -60,7 +60,7 @@ resource "digitalocean_tag" "workers" {
 data "ct_config" "worker-ignition" {
  content      = data.template_file.worker-config.rendered
  pretty_print = false
-  snippets     = var.worker_clc_snippets
+  snippets     = var.worker_snippets
 }

 # Worker Container Linux config
--- a/digital-ocean/fedora-coreos/kubernetes/LICENSE
+++ b/digital-ocean/fedora-coreos/kubernetes/LICENSE
@ -0,0 +1,23 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Typhoon Authors
+Copyright (c) 2020 Dalton Hubble
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
--- a/digital-ocean/fedora-coreos/kubernetes/README.md
+++ b/digital-ocean/fedora-coreos/kubernetes/README.md
@ -0,0 +1,23 @@
+# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+
+Typhoon is a minimal and free Kubernetes distribution.
+
+* Minimal, stable base Kubernetes distribution
+* Declarative infrastructure and configuration
+* Free (freedom and cost) and privacy-respecting
+* Practical for labs, datacenters, and clouds
+
+Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
+
+## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
+
+* Kubernetes v1.18.1 (upstream)
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+* Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/) customization
+* Ready for Ingress, Prometheus, Grafana, CSI, and other [addons](https://typhoon.psdn.io/addons/overview/)
+
+## Docs
+
+Please see the [official docs](https://typhoon.psdn.io) and the Digital Ocean [tutorial](https://typhoon.psdn.io/fedora-coreos/digitalocean/).
+
--- a/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/bootstrap.tf
@ -0,0 +1,25 @@
+# Kubernetes assets (kubeconfig, manifests)
+module "bootstrap" {
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"
+
+  cluster_name = var.cluster_name
+  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
+  etcd_servers = digitalocean_record.etcds.*.fqdn
+  asset_dir    = var.asset_dir
+
+  networking = var.networking
+
+  # only effective with Calico networking
+  network_encapsulation = "vxlan"
+  network_mtu           = "1450"
+
+  pod_cidr              = var.pod_cidr
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  enable_reporting      = var.enable_reporting
+  enable_aggregation    = var.enable_aggregation
+
+  # Fedora CoreOS
+  trusted_certs_dir = "/etc/pki/tls/certs"
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/controllers.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/controllers.tf
@ -0,0 +1,100 @@
+# Controller Instance DNS records
+resource "digitalocean_record" "controllers" {
+  count = var.controller_count
+
+  # DNS zone where record should be created
+  domain = var.dns_zone
+
+  # DNS record (will be prepended to domain)
+  name = var.cluster_name
+  type = "A"
+  ttl  = 300
+
+  # IPv4 addresses of controllers
+  value = digitalocean_droplet.controllers.*.ipv4_address[count.index]
+}
+
+# Discrete DNS records for each controller's private IPv4 for etcd usage
+resource "digitalocean_record" "etcds" {
+  count = var.controller_count
+
+  # DNS zone where record should be created
+  domain = var.dns_zone
+
+  # DNS record (will be prepended to domain)
+  name = "${var.cluster_name}-etcd${count.index}"
+  type = "A"
+  ttl  = 300
+
+  # private IPv4 address for etcd
+  value = digitalocean_droplet.controllers.*.ipv4_address_private[count.index]
+}
+
+# Controller droplet instances
+resource "digitalocean_droplet" "controllers" {
+  count = var.controller_count
+
+  name   = "${var.cluster_name}-controller-${count.index}"
+  region = var.region
+
+  image = var.os_image
+  size  = var.controller_type
+
+  # network
+  # TODO: Only official DigitalOcean images support IPv6
+  ipv6               = false
+  private_networking = true
+
+  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]
+  ssh_keys  = var.ssh_fingerprints
+
+  tags = [
+    digitalocean_tag.controllers.id,
+  ]
+
+  lifecycle {
+    ignore_changes = [user_data]
+  }
+}
+
+# Tag to label controllers
+resource "digitalocean_tag" "controllers" {
+  name = "${var.cluster_name}-controller"
+}
+
+# Controller Ignition configs
+data "ct_config" "controller-ignitions" {
+  count        = var.controller_count
+  content      = data.template_file.controller-configs.*.rendered[count.index]
+  strict       = true
+  snippets     = var.controller_snippets
+}
+
+# Controller Fedora CoreOS configs
+data "template_file" "controller-configs" {
+  count = var.controller_count
+
+  template = file("${path.module}/fcc/controller.yaml")
+
+  vars = {
+    # Cannot use cyclic dependencies on controllers or their DNS records
+    etcd_name   = "etcd${count.index}"
+    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
+    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+  }
+}
+
+data "template_file" "etcds" {
+  count    = var.controller_count
+  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
+
+  vars = {
+    index        = count.index
+    cluster_name = var.cluster_name
+    dns_zone     = var.dns_zone
+  }
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/controller.yaml
@ -0,0 +1,214 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/coreos/etcd
+        Wants=network-online.target network.target
+        After=network-online.target
+        [Service]
+        # https://github.com/opencontainers/runc/pull/1807
+        # Type=notify
+        # NotifyAccess=exec
+        Type=exec
+        Restart=on-failure
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        ExecStartPre=/bin/mkdir -p /var/lib/etcd
+        ExecStartPre=-/usr/bin/podman rm etcd
+        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
+        ExecStart=/usr/bin/podman run --name etcd \
+          --env-file /etc/etcd/etcd.env \
+          --network host \
+          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
+          quay.io/coreos/etcd:v3.4.7
+        ExecStop=/usr/bin/podman stop etcd
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube (System Container)
+        Requires=afterburn.service
+        After=afterburn.service
+        Wants=rpc-statd.service
+        [Service]
+        EnvironmentFile=/run/metadata/afterburn
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/podman rm kubelet
+        ExecStart=/usr/bin/podman run --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /usr/lib/os-release:/etc/os-release:ro \
+          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /lib/modules:/lib/modules:ro \
+          --volume /run:/run \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /var/lib/calico:/var/lib/calico:ro \
+          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+          --volume /var/log:/var/log \
+          --volume /var/run/lock:/var/run/lock:z \
+          --volume /opt/cni/bin:/opt/cni/bin:z \
+          quay.io/poseidon/kubelet:v1.18.1 \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --cgroup-driver=systemd \
+          --cgroups-per-qos=true \
+          --enforce-node-allocatable=pods \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --healthz-port=0 \
+          --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/master \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/podman stop kubelet
+        Delegate=yes
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: kubelet.path
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Watch for kubeconfig
+        [Path]
+        PathExists=/etc/kubernetes/kubeconfig
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        ExecStartPre=-/usr/bin/podman rm bootstrap
+        ExecStart=/usr/bin/podman run --name bootstrap \
+            --network host \
+            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,Z \
+            --volume /opt/bootstrap/assets:/assets:ro,Z \
+            --volume /opt/bootstrap/apply:/apply:ro,Z \
+            --entrypoint=/apply \
+            quay.io/poseidon/kubelet:v1.18.1
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        ExecStartPost=-/usr/bin/podman stop bootstrap
+storage:
+  directories:
+    - path: /etc/kubernetes
+    - path: /opt/bootstrap
+  files:
+    - path: /opt/bootstrap/layout
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
+          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          sudo mkdir -p /etc/kubernetes/manifests
+          sudo mv static-manifests/* /etc/kubernetes/manifests/
+          sudo mkdir -p /opt/bootstrap/assets
+          sudo mv manifests /opt/bootstrap/assets/manifests
+          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests-networking
+    - path: /opt/bootstrap/apply
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/sysctl.d/max-user-watches.conf
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/systemd/system.conf.d/accounting.conf
+      contents:
+        inline: |
+          [Manager]
+          DefaultCPUAccounting=yes
+          DefaultMemoryAccounting=yes
+          DefaultBlockIOAccounting=yes
+    - path: /etc/etcd/etcd.env
+      mode: 0644
+      contents:
+        inline: |
+          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
+          NOTIFY_SOCKET=/run/systemd/notify
+          ETCD_NAME=${etcd_name}
+          ETCD_DATA_DIR=/var/lib/etcd
+          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+          ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+          ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+          ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+          ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+          ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+          ETCD_STRICT_RECONFIG_CHECK=true
+          ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+          ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+          ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+          ETCD_CLIENT_CERT_AUTH=true
+          ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+          ETCD_PEER_CLIENT_CERT_AUTH=true
--- a/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
+++ b/digital-ocean/fedora-coreos/kubernetes/fcc/worker.yaml
@ -0,0 +1,117 @@
+---
+variant: fcos
+version: 1.0.0
+systemd:
+  units:
+    - name: docker.service
+      enabled: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet via Hyperkube (System Container)
+        Requires=afterburn.service
+        After=afterburn.service
+        Wants=rpc-statd.service
+        [Service]
+        EnvironmentFile=/run/metadata/afterburn
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=-/usr/bin/podman rm kubelet
+        ExecStart=/usr/bin/podman run --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /usr/lib/os-release:/etc/os-release:ro \
+          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /lib/modules:/lib/modules:ro \
+          --volume /run:/run \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
+          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
+          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /var/lib/calico:/var/lib/calico:ro \
+          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
+          --volume /var/log:/var/log \
+          --volume /var/run/lock:/var/run/lock:z \
+          --volume /opt/cni/bin:/opt/cni/bin:z \
+          quay.io/poseidon/kubelet:v1.18.1 \
+          --anonymous-auth=false \
+          --authentication-token-webhook \
+          --authorization-mode=Webhook \
+          --cgroup-driver=systemd \
+          --cgroups-per-qos=true \
+          --enforce-node-allocatable=pods \
+          --client-ca-file=/etc/kubernetes/ca.crt \
+          --cluster_dns=${cluster_dns_service_ip} \
+          --cluster_domain=${cluster_domain_suffix} \
+          --cni-conf-dir=/etc/kubernetes/cni/net.d \
+          --exit-on-lock-contention \
+          --healthz-port=0 \
+          --hostname-override=$${AFTERBURN_DIGITALOCEAN_IPV4_PRIVATE_0} \
+          --kubeconfig=/etc/kubernetes/kubeconfig \
+          --lock-file=/var/run/lock/kubelet.lock \
+          --network-plugin=cni \
+          --node-labels=node.kubernetes.io/node \
+          --pod-manifest-path=/etc/kubernetes/manifests \
+          --read-only-port=0 \
+          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+        ExecStop=-/usr/bin/podman stop kubelet
+        Delegate=yes
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: kubelet.path
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Watch for kubeconfig
+        [Path]
+        PathExists=/etc/kubernetes/kubeconfig
+        [Install]
+        WantedBy=multi-user.target
+    - name: delete-node.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Delete Kubernetes node on shutdown
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/true
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /etc/kubernetes
+  files:
+    - path: /etc/sysctl.d/max-user-watches.conf
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/systemd/system.conf.d/accounting.conf
+      contents:
+        inline: |
+          [Manager]
+          DefaultCPUAccounting=yes
+          DefaultMemoryAccounting=yes
+          DefaultBlockIOAccounting=yes
--- a/digital-ocean/fedora-coreos/kubernetes/network.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/network.tf
@ -0,0 +1,117 @@
+resource "digitalocean_firewall" "rules" {
+  name = var.cluster_name
+
+  tags = ["${var.cluster_name}-controller", "${var.cluster_name}-worker"]
+
+  # allow ssh, internal flannel, internal node-exporter, internal kubelet
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "22"
+    source_addresses = ["0.0.0.0/0", "::/0"]
+  }
+
+  inbound_rule {
+    protocol    = "udp"
+    port_range  = "4789"
+    source_tags = [digitalocean_tag.controllers.name, digitalocean_tag.workers.name]
+  }
+
+  # Allow Prometheus to scrape node-exporter
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "9100"
+    source_tags = [digitalocean_tag.workers.name]
+  }
+
+  # Allow Prometheus to scrape kube-proxy
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "10249"
+    source_tags = [digitalocean_tag.workers.name]
+  }
+
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "10250"
+    source_tags = [digitalocean_tag.controllers.name, digitalocean_tag.workers.name]
+  }
+
+  # allow all outbound traffic
+  outbound_rule {
+    protocol              = "tcp"
+    port_range            = "1-65535"
+    destination_addresses = ["0.0.0.0/0", "::/0"]
+  }
+
+  outbound_rule {
+    protocol              = "udp"
+    port_range            = "1-65535"
+    destination_addresses = ["0.0.0.0/0", "::/0"]
+  }
+
+  outbound_rule {
+    protocol              = "icmp"
+    port_range            = "1-65535"
+    destination_addresses = ["0.0.0.0/0", "::/0"]
+  }
+}
+
+resource "digitalocean_firewall" "controllers" {
+  name = "${var.cluster_name}-controllers"
+
+  tags = ["${var.cluster_name}-controller"]
+
+  # etcd
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "2379-2380"
+    source_tags = [digitalocean_tag.controllers.name]
+  }
+
+  # etcd metrics
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "2381"
+    source_tags = [digitalocean_tag.workers.name]
+  }
+
+  # kube-apiserver
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "6443"
+    source_addresses = ["0.0.0.0/0", "::/0"]
+  }
+
+  # kube-scheduler metrics, kube-controller-manager metrics
+  inbound_rule {
+    protocol    = "tcp"
+    port_range  = "10251-10252"
+    source_tags = [digitalocean_tag.workers.name]
+  }
+}
+
+resource "digitalocean_firewall" "workers" {
+  name = "${var.cluster_name}-workers"
+
+  tags = ["${var.cluster_name}-worker"]
+
+  # allow HTTP/HTTPS ingress
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "80"
+    source_addresses = ["0.0.0.0/0", "::/0"]
+  }
+
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "443"
+    source_addresses = ["0.0.0.0/0", "::/0"]
+  }
+
+  inbound_rule {
+    protocol         = "tcp"
+    port_range       = "10254"
+    source_addresses = ["0.0.0.0/0"]
+  }
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/outputs.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/outputs.tf
@ -0,0 +1,47 @@
+output "kubeconfig-admin" {
+  value = module.bootstrap.kubeconfig-admin
+}
+
+output "controllers_dns" {
+  value = digitalocean_record.controllers[0].fqdn
+}
+
+output "workers_dns" {
+  # Multiple A and AAAA records with the same FQDN
+  value = digitalocean_record.workers-record-a[0].fqdn
+}
+
+output "controllers_ipv4" {
+  value = digitalocean_droplet.controllers.*.ipv4_address
+}
+
+output "controllers_ipv6" {
+  value = digitalocean_droplet.controllers.*.ipv6_address
+}
+
+output "workers_ipv4" {
+  value = digitalocean_droplet.workers.*.ipv4_address
+}
+
+output "workers_ipv6" {
+  value = digitalocean_droplet.workers.*.ipv6_address
+}
+
+# Outputs for worker pools
+
+output "kubeconfig" {
+  value = module.bootstrap.kubeconfig-kubelet
+}
+
+# Outputs for custom firewalls
+
+output "controller_tag" {
+  description = "Tag applied to controller droplets"
+  value       = digitalocean_tag.controllers.name
+}
+
+output "worker_tag" {
+  description = "Tag applied to worker droplets"
+  value       = digitalocean_tag.workers.name
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/ssh.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/ssh.tf
@ -0,0 +1,87 @@
+locals {
+  # format assets for distribution
+  assets_bundle = [
+    # header with the unpack location
+    for key, value in module.bootstrap.assets_dist :
+    format("##### %s\n%s", key, value)
+  ]
+}
+
+# Secure copy assets to controllers. Activates kubelet.service
+resource "null_resource" "copy-controller-secrets" {
+  count = var.controller_count
+
+  depends_on = [
+    module.bootstrap,
+    digitalocean_firewall.rules
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = digitalocean_droplet.controllers.*.ipv4_address[count.index]
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = module.bootstrap.kubeconfig-kubelet
+    destination = "$HOME/kubeconfig"
+  }
+
+  provisioner "file" {
+    content     = join("\n", local.assets_bundle)
+    destination = "$HOME/assets"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+      "sudo /opt/bootstrap/layout",
+    ]
+  }
+}
+
+# Secure copy kubeconfig to all workers. Activates kubelet.service.
+resource "null_resource" "copy-worker-secrets" {
+  count = var.worker_count
+
+  connection {
+    type    = "ssh"
+    host    = digitalocean_droplet.workers.*.ipv4_address[count.index]
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "file" {
+    content     = module.bootstrap.kubeconfig-kubelet
+    destination = "$HOME/kubeconfig"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo mv $HOME/kubeconfig /etc/kubernetes/kubeconfig",
+    ]
+  }
+}
+
+# Connect to a controller to perform one-time cluster bootstrap.
+resource "null_resource" "bootstrap" {
+  depends_on = [
+    null_resource.copy-controller-secrets,
+    null_resource.copy-worker-secrets,
+  ]
+
+  connection {
+    type    = "ssh"
+    host    = digitalocean_droplet.controllers[0].ipv4_address
+    user    = "core"
+    timeout = "15m"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo systemctl start bootstrap",
+    ]
+  }
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/variables.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/variables.tf
@ -0,0 +1,114 @@
+variable "cluster_name" {
+  type        = string
+  description = "Unique cluster name (prepended to dns_zone)"
+}
+
+# Digital Ocean
+
+variable "region" {
+  type        = string
+  description = "Digital Ocean region (e.g. nyc1, sfo2, fra1, tor1)"
+}
+
+variable "dns_zone" {
+  type        = string
+  description = "Digital Ocean domain (i.e. DNS zone) (e.g. do.example.com)"
+}
+
+# instances
+
+variable "controller_count" {
+  type        = number
+  description = "Number of controllers (i.e. masters)"
+  default     = 1
+}
+
+variable "worker_count" {
+  type        = number
+  description = "Number of workers"
+  default     = 1
+}
+
+variable "controller_type" {
+  type        = string
+  description = "Droplet type for controllers (e.g. s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb)."
+  default     = "s-2vcpu-2gb"
+}
+
+variable "worker_type" {
+  type        = string
+  description = "Droplet type for workers (e.g. s-1vcpu-2gb, s-2vcpu-2gb)"
+  default     = "s-1vcpu-2gb"
+}
+
+variable "os_image" {
+  type        = string
+  description = "Fedora CoreOS image for instances"
+}
+
+variable "controller_snippets" {
+  type        = list(string)
+  description = "Controller Fedora CoreOS Config snippets"
+  default     = []
+}
+
+variable "worker_snippets" {
+  type        = list(string)
+  description = "Worker Fedora CoreOS Config snippets"
+  default     = []
+}
+
+# configuration
+
+variable "ssh_fingerprints" {
+  type        = list(string)
+  description = "SSH public key fingerprints. (e.g. see `ssh-add -l -E md5`)"
+}
+
+variable "networking" {
+  type        = string
+  description = "Choice of networking provider (flannel or calico)"
+  default     = "calico"
+}
+
+variable "pod_cidr" {
+  type        = string
+  description = "CIDR IPv4 range to assign Kubernetes pods"
+  default     = "10.2.0.0/16"
+}
+
+variable "service_cidr" {
+  type        = string
+  description = <<EOD
+CIDR IPv4 range to assign Kubernetes services.
+The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
+EOD
+  default     = "10.3.0.0/16"
+}
+
+variable "enable_reporting" {
+  type        = bool
+  description = "Enable usage or analytics reporting to upstreams (Calico)"
+  default     = false
+}
+
+variable "enable_aggregation" {
+  type        = bool
+  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  default     = false
+}
+
+# unofficial, undocumented, unsupported
+
+variable "asset_dir" {
+  type        = string
+  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
+  default     = ""
+}
+
+variable "cluster_domain_suffix" {
+  type        = string
+  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
+  default     = "cluster.local"
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/versions.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/versions.tf
@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.6"
+  required_providers {
+    digitalocean = "~> 1.3"
+    ct           = "~> 0.3"
+    template     = "~> 2.1"
+    null         = "~> 2.1"
+  }
+}
+
--- a/digital-ocean/fedora-coreos/kubernetes/workers.tf
+++ b/digital-ocean/fedora-coreos/kubernetes/workers.tf
@ -0,0 +1,77 @@
+# Worker DNS records
+resource "digitalocean_record" "workers-record-a" {
+  count = var.worker_count
+
+  # DNS zone where record should be created
+  domain = var.dns_zone
+
+  name  = "${var.cluster_name}-workers"
+  type  = "A"
+  ttl   = 300
+  value = digitalocean_droplet.workers.*.ipv4_address[count.index]
+}
+
+/*
+# TODO: Only official DigitalOcean images support IPv6
+resource "digitalocean_record" "workers-record-aaaa" {
+  count = var.worker_count
+
+  # DNS zone where record should be created
+  domain = var.dns_zone
+
+  name  = "${var.cluster_name}-workers"
+  type  = "AAAA"
+  ttl   = 300
+  value = digitalocean_droplet.workers.*.ipv6_address[count.index]
+}
+*/
+
+# Worker droplet instances
+resource "digitalocean_droplet" "workers" {
+  count = var.worker_count
+
+  name   = "${var.cluster_name}-worker-${count.index}"
+  region = var.region
+
+  image = var.os_image
+  size  = var.worker_type
+
+  # network
+  # TODO: Only official DigitalOcean images support IPv6
+  ipv6               = false
+  private_networking = true
+
+  user_data = data.ct_config.worker-ignition.rendered
+  ssh_keys  = var.ssh_fingerprints
+
+  tags = [
+    digitalocean_tag.workers.id,
+  ]
+
+  lifecycle {
+    create_before_destroy = true
+  }
+}
+
+# Tag to label workers
+resource "digitalocean_tag" "workers" {
+  name = "${var.cluster_name}-worker"
+}
+
+# Worker Ignition config
+data "ct_config" "worker-ignition" {
+  content      = data.template_file.worker-config.rendered
+  strict       = true
+  snippets     = var.worker_snippets
+}
+
+# Worker Fedora CoreOS config
+data "template_file" "worker-config" {
+  template = file("${path.module}/fcc/worker.yaml")
+
+  vars = {
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+  }
+}
+
--- a/digital-ocean/ignore/.gitkeep
+++ b/digital-ocean/ignore/.gitkeep
--- a/docs/advanced/customization.md
+++ b/docs/advanced/customization.md
@ -125,7 +125,7 @@ systemd:
            Environment="ETCD_LOG_PACKAGE_LEVELS=etcdserver=WARNING,security=DEBUG"
 ```

-Reference the CLC contents by location (e.g. `file("./custom-units.yaml")`). On [AWS](/cl/aws/#cluster), [Azure](/cl/azure/#cluster), [DigitalOcean](/cl/digital-ocean/#cluster), or [Google Cloud](/cl/google-cloud/#cluster) extend the `controller_clc_snippets` or `worker_clc_snippets` list variables.
+Reference the CLC contents by location (e.g. `file("./custom-units.yaml")`). On [AWS](/cl/aws/#cluster), [Azure](/cl/azure/#cluster), [DigitalOcean](/cl/digital-ocean/#cluster), or [Google Cloud](/cl/google-cloud/#cluster) extend the `controller_snippets` or `worker_snippets` list variables.

 ```tf
 module "nemo" {
@ -133,11 +133,11 @@ module "nemo" {

  controller_count        = 1
  worker_count            = 2
-  controller_clc_snippets = [
+  controller_snippets = [
    file("./custom-files"),
    file("./custom-units"),
  ]
-  worker_clc_snippets = [
+  worker_snippets = [
    file("./custom-files"),
    file("./custom-units")",
  ]
@ -145,12 +145,12 @@ module "nemo" {
 }
 ```

-On [Bare-Metal](/cl/bare-metal/#cluster), different CLCs may be used for each node (since hardware may be heterogeneous). Extend the `clc_snippets` map variable by mapping a controller or worker name key to a list of snippets.
+On [Bare-Metal](/cl/bare-metal/#cluster), different CLCs may be used for each node (since hardware may be heterogeneous). Extend the `snippets` map variable by mapping a controller or worker name key to a list of snippets.

 ```tf
 module "mercury" {
  ...
-  clc_snippets = {
+  snippets = {
    "node2" = [file("./units/hello.yaml")]
    "node3" = [
      file("./units/world.yaml"),
--- a/docs/advanced/worker-pools.md
+++ b/docs/advanced/worker-pools.md
@ -67,7 +67,7 @@ The AWS internal `workers` module supports a number of [variables](https://githu
 | disk_type | Type of the EBS volume | "gp2" | standard, gp2, io1 |
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | spot_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0 | 0.10 |
-| clc_snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |

@ -79,7 +79,7 @@ Create a cluster following the Azure [tutorial](../cl/azure.md#cluster). Define

 ```tf
 module "ramius-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes/workers?ref=v1.18.1"
  
  # Azure
  region                  = module.ramius.region
@ -133,7 +133,7 @@ The Azure internal `workers` module supports a number of [variables](https://git
 | vm_type | Machine type for instances | "Standard_DS1_v2" | See below |
 | os_image | Channel for a Container Linux derivative | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha |
 | priority | Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | "Regular" | "Low" |
-| clc_snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |

@ -145,7 +145,7 @@ Create a cluster following the Google Cloud [tutorial](../cl/google-cloud.md#clu

 ```tf
 module "yavin-worker-pool" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes/workers?ref=v1.18.1"

  # Google Cloud
  region       = "europe-west2"
@ -176,11 +176,11 @@ Verify a managed instance group of workers joins the cluster within a few minute
 ```
 $ kubectl get nodes
 NAME                                             STATUS   AGE    VERSION
-yavin-controller-0.c.example-com.internal        Ready    6m     v1.18.0
-yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.18.0
-yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.18.0
-yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.18.0
-yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.18.0
+yavin-controller-0.c.example-com.internal        Ready    6m     v1.18.1
+yavin-worker-jrbf.c.example-com.internal         Ready    5m     v1.18.1
+yavin-worker-mzdm.c.example-com.internal         Ready    5m     v1.18.1
+yavin-16x-worker-jrbf.c.example-com.internal     Ready    3m     v1.18.1
+yavin-16x-worker-mzdm.c.example-com.internal     Ready    3m     v1.18.1
 ```

 ### Variables
@ -209,7 +209,7 @@ Check the list of regions [docs](https://cloud.google.com/compute/docs/regions-z
 | os_image | Container Linux image for compute instances | "coreos-stable" | "coreos-alpha", "coreos-beta" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | preemptible | If true, Compute Engine will terminate instances randomly within 24 hours | false | true |
-| clc_snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| snippets | Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | service_cidr | Must match `service_cidr` of cluster | "10.3.0.0/16" | "10.3.0.0/24" |
 | node_labels | List of initial node labels | [] | ["worker-pool=foo"] |

--- a/docs/cl/aws.md
+++ b/docs/cl/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.18.0 cluster on AWS with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on AWS with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/container-linux/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/container-linux/kubernetes?ref=v1.18.1"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES   AGE  VERSION
-ip-10-0-3-155  Ready   <none>  10m  v1.18.0
-ip-10-0-26-65  Ready   <none>  10m  v1.18.0
-ip-10-0-41-21  Ready   <none>  10m  v1.18.0
+ip-10-0-3-155  Ready   <none>  10m  v1.18.1
+ip-10-0-26-65  Ready   <none>  10m  v1.18.1
+ip-10-0-41-21  Ready   <none>  10m  v1.18.1
 ```

 List the pods.
@ -214,8 +214,8 @@ Reference the DNS zone id with `aws_route53_zone.zone-for-clusters.zone_id`.
 | disk_iops | IOPS of the EBS volume | 0 (i.e. auto) | 400 |
 | worker_target_groups | Target group ARNs to which worker instances should be added | [] | [aws_lb_target_group.app.id] |
 | worker_price | Spot price in USD for worker instances or 0 to use on-demand instances | 0/null | 0.10 |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico only) | 1480 | 8981 |
 | host_cidr | CIDR IPv4 range to assign to EC2 instances | "10.0.0.0/16" | "10.1.0.0/16" |
--- a/docs/cl/azure.md
+++ b/docs/cl/azure.md
@ -3,7 +3,7 @@
 !!! danger
    Typhoon for Azure is alpha. For production, use AWS, Google Cloud, or bare-metal. As Azure matures, check [errata](https://github.com/poseidon/typhoon/wiki/Errata) for known shortcomings.

-In this tutorial, we'll create a Kubernetes v1.18.0 cluster on Azure with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on Azure with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a resource group, virtual network, subnets, security groups, controller availability set, worker scale set, load balancer, and TLS assets.

@ -66,7 +66,7 @@ Define a Kubernetes cluster using the module `azure/container-linux/kubernetes`.

 ```tf
 module "ramius" {
-  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//azure/container-linux/kubernetes?ref=v1.18.1"

  # Azure
  cluster_name   = "ramius"
@ -149,9 +149,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/ramius-config
 $ kubectl get nodes
 NAME                  STATUS  ROLES   AGE  VERSION
-ramius-controller-0   Ready   <none>  24m  v1.18.0
-ramius-worker-000001  Ready   <none>  25m  v1.18.0
-ramius-worker-000002  Ready   <none>  24m  v1.18.0
+ramius-controller-0   Ready   <none>  24m  v1.18.1
+ramius-worker-000001  Ready   <none>  25m  v1.18.1
+ramius-worker-000002  Ready   <none>  24m  v1.18.1
 ```

 List the pods.
@ -231,8 +231,8 @@ Reference the DNS zone with `azurerm_dns_zone.clusters.name` and its resource gr
 | os_image | Channel for a Container Linux derivative | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_priority | Set priority to Spot to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time | Regular | Spot |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
+| worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/#usage) |
 | networking | Choice of networking provider | "calico" | "flannel" or "calico" |
 | host_cidr | CIDR IPv4 range to assign to instances | "10.0.0.0/16" | "10.0.0.0/20" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
--- a/docs/cl/bare-metal.md
+++ b/docs/cl/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.18.0 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll network boot and provision a Kubernetes v1.18.1 cluster on bare-metal with CoreOS Container Linux or Flatcar Linux.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Container Linux to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/container-linux/kuberne

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.1"

  # bare-metal
  cluster_name            = "mercury"
@ -299,9 +299,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.18.0
-node2.example.com   Ready   <none>  10m  v1.18.0
-node3.example.com   Ready   <none>  10m  v1.18.0
+node1.example.com   Ready   <none>  10m  v1.18.1
+node2.example.com   Ready   <none>  10m  v1.18.1
+node3.example.com   Ready   <none>  10m  v1.18.1
 ```

 List the pods.
@ -352,7 +352,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/bare-me
 | install_disk | Disk device where Container Linux should be installed | "/dev/sda" | "/dev/sdb" |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | network_mtu | CNI interface MTU (calico-only) | 1480 | - |
-| clc_snippets | Map from machine names to lists of Container Linux Config snippets | {} | [example](/advanced/customization/#usage) |
+| snippets | Map from machine names to lists of Container Linux Config snippets | {} | [examples](/advanced/customization/) |
 | network_ip_autodetection_method | Method to detect host IPv4 address (calico-only) | "first-found" | "can-reach=10.0.0.1" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
--- a/docs/cl/digital-ocean.md
+++ b/docs/cl/digital-ocean.md
@ -1,6 +1,6 @@
 # Digital Ocean

-In this tutorial, we'll create a Kubernetes v1.18.0 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on DigitalOcean with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.

@ -50,7 +50,7 @@ Configure the DigitalOcean provider to use your token in a `providers.tf` file.

 ```tf
 provider "digitalocean" {
-  version = "1.14.0"
+  version = "1.15.1"
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
 }

@ -65,7 +65,7 @@ Define a Kubernetes cluster using the module `digital-ocean/container-linux/kube

 ```tf
 module "nemo" {
-  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.18.1"

  # Digital Ocean
  cluster_name = "nemo"
@ -161,9 +161,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/nemo-config
 $ kubectl get nodes
 NAME               STATUS  ROLES   AGE  VERSION
-10.132.110.130     Ready   <none>  10m  v1.18.0
-10.132.115.81      Ready   <none>  10m  v1.18.0
-10.132.124.107     Ready   <none>  10m  v1.18.0
+10.132.110.130     Ready   <none>  10m  v1.18.1
+10.132.115.81      Ready   <none>  10m  v1.18.1
+10.132.124.107     Ready   <none>  10m  v1.18.1
 ```

 List the pods.
@ -204,7 +204,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/digital

 Clusters create DNS A records `${cluster_name}.${dns_zone}` to resolve to controller droplets (round robin). This FQDN is used by workers and `kubectl` to access the apiserver(s). In this example, the cluster's apiserver would be accessible at `nemo.do.example.com`.

-You'll need a registered domain name or delegated subdomain in Digital Ocean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.
+You'll need a registered domain name or delegated subdomain in DigitalOcean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.

 ```tf
 # Declare a DigitalOcean record to also create a zone file
@ -244,8 +244,8 @@ Digital Ocean requires the SSH public key be uploaded to your account, so you ma
 | controller_type | Droplet type for controllers | "s-2vcpu-2gb" | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
 | worker_type | Droplet type for workers | "s-1vcpu-2gb" | s-1vcpu-2gb, s-2vcpu-2gb, ... |
 | os_image | Container Linux image for instances | "coreos-stable" | coreos-stable, coreos-beta, coreos-alpha, "custom-image-id" |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "flannel" or "calico" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
--- a/docs/cl/google-cloud.md
+++ b/docs/cl/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.18.0 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on Google Compute Engine with CoreOS Container Linux or Flatcar Linux.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -71,7 +71,7 @@ Define a Kubernetes cluster using the module `google-cloud/container-linux/kuber

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -167,9 +167,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
 ```

 List the pods.
@ -238,8 +238,8 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | os_image | Container Linux image for compute instances | "coreos-stable" | "flatcar-linux-2303-4-0" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
-| controller_clc_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
-| worker_clc_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| controller_snippets | Controller Container Linux Config snippets | [] | [example](/advanced/customization/) |
+| worker_snippets | Worker Container Linux Config snippets | [] | [example](/advanced/customization/) |
 | networking | Choice of networking provider | "calico" | "calico" or "flannel" |
 | pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
 | service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
--- a/docs/fedora-coreos/aws.md
+++ b/docs/fedora-coreos/aws.md
@ -1,6 +1,6 @@
 # AWS

-In this tutorial, we'll create a Kubernetes v1.18.0 cluster on AWS with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on AWS with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a VPC, gateway, subnets, security groups, controller instances, worker auto-scaling group, network load balancer, and TLS assets.

@ -70,7 +70,7 @@ Define a Kubernetes cluster using the module `aws/fedora-coreos/kubernetes`.

 ```tf
 module "tempest" {
-  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.18.1"

  # AWS
  cluster_name = "tempest"
@ -143,9 +143,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/tempest-config
 $ kubectl get nodes
 NAME           STATUS  ROLES    AGE  VERSION
-ip-10-0-3-155  Ready   <none>   10m  v1.18.0
-ip-10-0-26-65  Ready   <none>   10m  v1.18.0
-ip-10-0-41-21  Ready   <none>   10m  v1.18.0
+ip-10-0-3-155  Ready   <none>   10m  v1.18.1
+ip-10-0-26-65  Ready   <none>   10m  v1.18.1
+ip-10-0-41-21  Ready   <none>   10m  v1.18.1
 ```

 List the pods.
--- a/docs/fedora-coreos/bare-metal.md
+++ b/docs/fedora-coreos/bare-metal.md
@ -1,6 +1,6 @@
 # Bare-Metal

-In this tutorial, we'll network boot and provision a Kubernetes v1.18.0 cluster on bare-metal with Fedora CoreOS.
+In this tutorial, we'll network boot and provision a Kubernetes v1.18.1 cluster on bare-metal with Fedora CoreOS.

 First, we'll deploy a [Matchbox](https://github.com/poseidon/matchbox) service and setup a network boot environment. Then, we'll declare a Kubernetes cluster using the Typhoon Terraform module and power on machines. On PXE boot, machines will install Fedora CoreOS to disk, reboot into the disk install, and provision themselves as Kubernetes controllers or workers via Ignition.

@ -160,7 +160,7 @@ Define a Kubernetes cluster using the module `bare-metal/fedora-coreos/kubernete

 ```tf
 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.1"

  # bare-metal
  cluster_name            = "mercury"
@ -289,9 +289,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/mercury-config
 $ kubectl get nodes
 NAME                STATUS  ROLES   AGE  VERSION
-node1.example.com   Ready   <none>  10m  v1.18.0
-node2.example.com   Ready   <none>  10m  v1.18.0
-node3.example.com   Ready   <none>  10m  v1.18.0
+node1.example.com   Ready   <none>  10m  v1.18.1
+node2.example.com   Ready   <none>  10m  v1.18.1
+node3.example.com   Ready   <none>  10m  v1.18.1
 ```

 List the pods.
--- a/docs/fedora-coreos/digitalocean.md
+++ b/docs/fedora-coreos/digitalocean.md
@ -0,0 +1,247 @@
+# Digital Ocean
+
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on DigitalOcean with Fedora CoreOS.
+
+We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create controller droplets, worker droplets, DNS records, tags, and TLS assets.
+
+Controller hosts are provisioned to run an `etcd-member` peer and a `kubelet` service. Worker hosts run a `kubelet` service. Controller nodes run `kube-apiserver`, `kube-scheduler`, `kube-controller-manager`, and `coredns`, while `kube-proxy` and `calico` (or `flannel`) run on every node. A generated `kubeconfig` provides `kubectl` access to the cluster.
+
+## Requirements
+
+* Digital Ocean Account and Token
+* Digital Ocean Domain (registered Domain Name or delegated subdomain)
+* Terraform v0.12.6+ and [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) installed locally
+
+## Terraform Setup
+
+Install [Terraform](https://www.terraform.io/downloads.html) v0.12.6+ on your system.
+
+```sh
+$ terraform version
+Terraform v0.12.21
+```
+
+Add the [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) plugin binary for your system to `~/.terraform.d/plugins/`, noting the final name.
+
+```sh
+wget https://github.com/poseidon/terraform-provider-ct/releases/download/v0.5.0/terraform-provider-ct-v0.5.0-linux-amd64.tar.gz
+tar xzf terraform-provider-ct-v0.5.0-linux-amd64.tar.gz
+mv terraform-provider-ct-v0.5.0-linux-amd64/terraform-provider-ct ~/.terraform.d/plugins/terraform-provider-ct_v0.5.0
+```
+
+Read [concepts](/architecture/concepts/) to learn about Terraform, modules, and organizing resources. Change to your infrastructure repository (e.g. `infra`).
+
+```
+cd infra/clusters
+```
+
+## Provider
+
+Login to [DigitalOcean](https://cloud.digitalocean.com) or create an [account](https://cloud.digitalocean.com/registrations/new), if you don't have one.
+
+Generate a Personal Access Token with read/write scope from the [API tab](https://cloud.digitalocean.com/settings/api/tokens). Write the token to a file that can be referenced in configs.
+
+```sh
+mkdir -p ~/.config/digital-ocean
+echo "TOKEN" > ~/.config/digital-ocean/token
+```
+
+Configure the DigitalOcean provider to use your token in a `providers.tf` file.
+
+```tf
+provider "digitalocean" {
+  version = "1.15.1"
+  token = "${chomp(file("~/.config/digital-ocean/token"))}"
+}
+
+provider "ct" {
+  version = "0.5.0"
+}
+```
+
+## Fedora CoreOS Images
+
+Fedora CoreOS publishes images for DigitalOcean, but does not yet upload them. DigitalOcean allows [custom images](https://blog.digitalocean.com/custom-images/) to be uploaded via URL or file.
+
+Import a [Fedora CoreOS](https://getfedora.org/en/coreos/download?tab=cloud_operators&stream=stable) image via URL to desired a region(s). Reference the DigitalOcean image and set the `os_image` in the next step.
+
+```tf
+data "digitalocean_image" "fedora-coreos-31-20200323-3-2" {
+  name = "fedora-coreos-31.20200323.3.2-digitalocean.x86_64.qcow2.gz"
+}
+```
+
+## Cluster
+
+Define a Kubernetes cluster using the module `digital-ocean/fedora-coreos/kubernetes`.
+
+```tf
+module "nemo" {
+  source = "git::https://github.com/poseidon/typhoon//digital-ocean/fedora-coreos/kubernetes?ref=v1.18.1"
+
+  # Digital Ocean
+  cluster_name = "nemo"
+  region       = "nyc3"
+  dns_zone     = "digital-ocean.example.com"
+  os_image     = data.digitalocean_image.fedora-coreos-31-20200323-3-2.id
+
+  # configuration
+  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]
+
+  # optional
+  worker_count = 2
+}
+```
+
+Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/digital-ocean/fedora-coreos/kubernetes/variables.tf) source.
+
+## ssh-agent
+
+Initial bootstrapping requires `bootstrap.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`.
+
+```sh
+ssh-add ~/.ssh/id_rsa
+ssh-add -L
+```
+
+## Apply
+
+Initialize the config directory if this is the first use with Terraform.
+
+```sh
+terraform init
+```
+
+Plan the resources to be created.
+
+```sh
+$ terraform plan
+Plan: 54 to add, 0 to change, 0 to destroy.
+```
+
+Apply the changes to create the cluster.
+
+```sh
+$ terraform apply
+module.nemo.null_resource.bootstrap: Still creating... (30s elapsed)
+module.nemo.null_resource.bootstrap: Provisioning with 'remote-exec'...
+...
+module.nemo.null_resource.bootstrap: Still creating... (6m20s elapsed)
+module.nemo.null_resource.bootstrap: Creation complete (ID: 7599298447329218468)
+
+Apply complete! Resources: 42 added, 0 changed, 0 destroyed.
+```
+
+In 3-6 minutes, the Kubernetes cluster will be ready.
+
+## Verify
+
+[Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your system. Obtain the generated cluster `kubeconfig` from module outputs (e.g. write to a local file).
+
+```
+resource "local_file" "kubeconfig-nemo" {
+  content  = module.nemo.kubeconfig-admin
+  filename = "/home/user/.kube/configs/nemo-config"
+}
+```
+
+List nodes in the cluster.
+
+```
+$ export KUBECONFIG=/home/user/.kube/configs/nemo-config
+$ kubectl get nodes
+NAME               STATUS  ROLES   AGE  VERSION
+10.132.110.130     Ready   <none>  10m  v1.18.1
+10.132.115.81      Ready   <none>  10m  v1.18.1
+10.132.124.107     Ready   <none>  10m  v1.18.1
+```
+
+List the pods.
+
+```
+NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
+kube-system   coredns-1187388186-ld1j7                   1/1       Running   0          11m
+kube-system   coredns-1187388186-rdhf7                   1/1       Running   0          11m
+kube-system   calico-node-1m5bf                          2/2       Running   0          11m
+kube-system   calico-node-7jmr1                          2/2       Running   0          11m
+kube-system   calico-node-bknc8                          2/2       Running   0          11m
+kube-system   kube-apiserver-ip-10.132.115.81            1/1       Running   0          11m
+kube-system   kube-controller-manager-ip-10.132.115.81   1/1       Running   0          11m
+kube-system   kube-proxy-6kxjf                           1/1       Running   0          11m
+kube-system   kube-proxy-fh3td                           1/1       Running   0          11m
+kube-system   kube-proxy-k35rc                           1/1       Running   0          11m
+kube-system   kube-scheduler-ip-10.132.115.81            1/1       Running   0          11m
+```
+
+## Going Further
+
+Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).
+
+## Variables
+
+Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/digital-ocean/fedora-coreos/kubernetes/variables.tf) source.
+
+### Required
+
+| Name | Description | Example |
+|:-----|:------------|:--------|
+| cluster_name | Unique cluster name (prepended to dns_zone) | "nemo" |
+| region | Digital Ocean region | "nyc1", "sfo2", "fra1", tor1" |
+| dns_zone | Digital Ocean domain (i.e. DNS zone) | "do.example.com" |
+| os_image | Fedora CoreOS image for instances | "custom-image-id" |
+| ssh_fingerprints | SSH public key fingerprints | ["d7:9d..."] |
+
+#### DNS Zone
+
+Clusters create DNS A records `${cluster_name}.${dns_zone}` to resolve to controller droplets (round robin). This FQDN is used by workers and `kubectl` to access the apiserver(s). In this example, the cluster's apiserver would be accessible at `nemo.do.example.com`.
+
+You'll need a registered domain name or delegated subdomain in DigitalOcean Domains (i.e. DNS zones). You can set this up once and create many clusters with unique names.
+
+```tf
+# Declare a DigitalOcean record to also create a zone file
+resource "digitalocean_domain" "zone-for-clusters" {
+  name       = "do.example.com"
+  ip_address = "8.8.8.8"
+}
+```
+
+!!! tip ""
+    If you have an existing domain name with a zone file elsewhere, just delegate a subdomain that can be managed on DigitalOcean (e.g. do.mydomain.com) and [update nameservers](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-host-name-with-digitalocean).
+
+#### SSH Fingerprints
+
+DigitalOcean droplets are created with your SSH public key "fingerprint" (i.e. MD5 hash) to allow access. If your SSH public key is at `~/.ssh/id_rsa`, find the fingerprint with,
+
+```bash
+ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub | awk '{print $2}'
+MD5:d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7
+```
+
+If you use `ssh-agent` (e.g. Yubikey for SSH), find the fingerprint with,
+
+```
+ssh-add -l -E md5
+2048 MD5:d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7 cardno:000603633110 (RSA)
+```
+
+Digital Ocean requires the SSH public key be uploaded to your account, so you may also find the fingerprint under Settings -> Security. Finally, if you don't have an SSH key, [create one now](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/).
+
+### Optional
+
+| Name | Description | Default | Example |
+|:-----|:------------|:--------|:--------|
+| controller_count | Number of controllers (i.e. masters) | 1 | 1 |
+| worker_count | Number of workers | 1 | 3 |
+| controller_type | Droplet type for controllers | "s-2vcpu-2gb" | s-2vcpu-2gb, s-2vcpu-4gb, s-4vcpu-8gb, ... |
+| worker_type | Droplet type for workers | "s-1vcpu-2gb" | s-1vcpu-2gb, s-2vcpu-2gb, ... |
+| controller_snippets | Controller Fedora CoreOS Config snippets | [] | [example](/advanced/customization/) |
+| worker_snippets | Worker Fedora CoreOS Config snippets | [] | [example](/advanced/customization/) |
+| networking | Choice of networking provider | "calico" | "flannel" or "calico" |
+| pod_cidr | CIDR IPv4 range to assign to Kubernetes pods | "10.2.0.0/16" | "10.22.0.0/16" |
+| service_cidr | CIDR IPv4 range to assign to Kubernetes services | "10.3.0.0/16" | "10.3.0.0/24" |
+
+Check the list of valid [droplet types](https://developers.digitalocean.com/documentation/changelog/api-v2/new-size-slugs-for-droplet-plan-changes/) or use `doctl compute size list`.
+
+!!! warning
+    Do not choose a `controller_type` smaller than 2GB. Smaller droplets are not sufficient for running a controller and bootstrapping will fail.
+
--- a/docs/fedora-coreos/google-cloud.md
+++ b/docs/fedora-coreos/google-cloud.md
@ -1,6 +1,6 @@
 # Google Cloud

-In this tutorial, we'll create a Kubernetes v1.18.0 cluster on Google Compute Engine with Fedora CoreOS.
+In this tutorial, we'll create a Kubernetes v1.18.1 cluster on Google Compute Engine with Fedora CoreOS.

 We'll declare a Kubernetes cluster using the Typhoon Terraform module. Then apply the changes to create a network, firewall rules, health checks, controller instances, worker managed instance group, load balancers, and TLS assets.

@ -73,13 +73,13 @@ Fedora CoreOS publishes images for Google Cloud, but does not yet upload them. G

 ```
 gsutil list
-gsutil cp fedora-coreos-31.20200310.3.0-gcp.x86_64.tar.gz gs://BUCKET
+gsutil cp fedora-coreos-31.20200323.3.2-gcp.x86_64.tar.gz gs://BUCKET
 ```

 Create a Compute Engine image from the file.

 ```
-gcloud compute images create fedora-coreos-31-20200310-3-0 --source-uri gs://BUCKET/fedora-coreos-31.20200310.3.0-gcp.x86_64.tar.gz
+gcloud compute images create fedora-coreos-31-20200323-3-2 --source-uri gs://BUCKET/fedora-coreos-31.20200323.3.2-gcp.x86_64.tar.gz
 ```

 ## Cluster
@ -97,7 +97,7 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # custom image name from above
-  os_image = "fedora-coreos-31-20200310-3-0"
+  os_image = "fedora-coreos-31-20200323-3-2"

  # configuration
  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
@ -107,7 +107,7 @@ module "yavin" {
 }
 ```

-Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.
+Reference the [variables docs](#variables) or the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/fedora-coreos/kubernetes/variables.tf) source.

 ## ssh-agent

@ -165,9 +165,9 @@ List nodes in the cluster.
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
 ```

 List the pods.
@ -194,7 +194,7 @@ Learn about [maintenance](/topics/maintenance/) and [addons](/addons/overview/).

 ## Variables

-Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/container-linux/kubernetes/variables.tf) source.
+Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-cloud/fedora-coreos/kubernetes/variables.tf) source.

 ### Required

@ -204,6 +204,7 @@ Check the [variables.tf](https://github.com/poseidon/typhoon/blob/master/google-
 | region | Google Cloud region | "us-central1" |
 | dns_zone | Google Cloud DNS zone | "google-cloud.example.com" |
 | dns_zone_name | Google Cloud DNS zone name | "example-zone" |
+| os_image | Fedora CoreOS image for compute instances | "fedora-coreos-31-20200323-3-2" |
 | ssh_authorized_key | SSH public key for user 'core' | "ssh-rsa AAAAB3NZ..." |

 Check the list of valid [regions](https://cloud.google.com/compute/docs/regions-zones/regions-zones) and list Fedora CoreOS [images](https://cloud.google.com/compute/docs/images) with `gcloud compute images list | grep fedora-coreos`.
@ -233,7 +234,6 @@ resource "google_dns_managed_zone" "zone-for-clusters" {
 | worker_count | Number of workers | 1 | 3 |
 | controller_type | Machine type for controllers | "n1-standard-1" | See below |
 | worker_type | Machine type for workers | "n1-standard-1" | See below |
-| os_image | Fedora CoreOS image for compute instances | "" | "fedora-coreos-31-20200113-3-1" |
 | disk_size | Size of the disk in GB | 40 | 100 |
 | worker_preemptible | If enabled, Compute Engine will terminate workers randomly within 24 hours | false | true |
 | controller_snippets | Controller Fedora CoreOS Config snippets | [] | [examples](/advanced/customization/) |
--- a/docs/index.md
+++ b/docs/index.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](advanced/worker-pools/), [preemptible](cl/google-cloud/#preemption) workers, and [snippets](advanced/customization/#container-linux) customization
@ -21,20 +21,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 Typhoon provides a Terraform Module for each supported operating system and platform.

-| Platform      | Operating System | Terraform Module | Status |
-|---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
-| Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
-| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
-| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |
-
 Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](fedora-coreos/aws.md) | stable |
 | Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](fedora-coreos/bare-metal.md) | beta |
+| DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](fedora-coreos/digitalocean.md) | alpha |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | beta |

 Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org/releases/).
@ -44,13 +37,24 @@ Typhoon is available for [Flatcar Container Linux](https://www.flatcar-linux.org
 | AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](cl/aws.md) | stable |
 | Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
 | Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
+| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | alpha |
 | Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | alpha |
-| Digital Ocean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | alpha |
+
+Typhoon is available for CoreOS Container Linux ([no updates](https://coreos.com/os/eol/) after May 2020).
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Container Linux  | [aws/container-linux/kubernetes](cl/aws.md) | stable |
+| Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
+| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](cl/bare-metal.md) | stable |
+| Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](cl/digital-ocean.md) | beta |
+| Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](cl/google-cloud.md) | stable |
+

 ## Documentation

 * Architecture [concepts](architecture/concepts.md) and [operating-systems](architecture/operating-systems.md)
-* Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Bare-Metal](fedora-coreos/bare-metal.md), and [Google Cloud](fedora-coreos/google-cloud.md)
+* Fedora CoreOS tutorials for [AWS](fedora-coreos/aws.md), [Bare-Metal](fedora-coreos/bare-metal.md), [DigitalOcean](fedora-coreos/digitalocean.md), and [Google Cloud](fedora-coreos/google-cloud.md)
 * Flatcar Linux tutorials for [AWS](cl/aws.md), [Azure](cl/azure.md), [Bare-Metal](cl/bare-metal.md), [DigitalOcean](cl/digital-ocean.md), and [Google Cloud](cl/google-cloud.md)

 ## Example
@ -59,7 +63,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.18.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -97,9 +101,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.0
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.0
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.0
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.1
 ```

 List the pods.
--- a/docs/topics/maintenance.md
+++ b/docs/topics/maintenance.md
@ -18,7 +18,7 @@ module "yavin" {
 }

 module "mercury" {
-  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.0"
+  source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.18.1"
  ...
 }
 ```
@ -279,15 +279,15 @@ Typhoon modules have been adapted for Terraform v0.12. Provider plugins requirem

 | Typhoon Release   | Terraform version   |
 |-------------------|---------------------|
-| v1.18.0 - ?       | v0.12.x             |
-| v1.10.3 - v1.18.0 | v0.11.x             |
+| v1.18.1 - ?       | v0.12.x             |
+| v1.10.3 - v1.18.1 | v0.11.x             |
 | v1.9.2 - v1.10.2  | v0.10.4+ or v0.11.x |
 | v1.7.3 - v1.9.1   | v0.10.x             |
 | v1.6.4 - v1.7.2   | v0.9.x              |

 ### New users

-New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.18.0+ without issue.
+New users can start with Terraform v0.12.x and follow the docs for Typhoon v1.18.1+ without issue.

 ### Existing users

--- a/google-cloud/container-linux/kubernetes/README.md
+++ b/google-cloud/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/google-cloud/container-linux/kubernetes/bootstrap.tf
+++ b/google-cloud/container-linux/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/container-linux/kubernetes/cl/controller.yaml
+++ b/google-cloud/container-linux/kubernetes/cl/controller.yaml
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.5"
+            Environment="ETCD_IMAGE_TAG=v3.4.7"
            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
            Environment="RKT_RUN_ARGS=--insecure-options=image"
            Environment="ETCD_NAME=${etcd_name}"
@ -90,7 +90,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -132,7 +132,7 @@ systemd:
            --volume script,kind=host,source=/opt/bootstrap/apply \
            --mount volume=script,target=/apply \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/apply
@ -168,9 +168,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      filesystem: root
      mode: 0544
--- a/google-cloud/container-linux/kubernetes/controllers.tf
+++ b/google-cloud/container-linux/kubernetes/controllers.tf
@ -68,7 +68,7 @@ data "ct_config" "controller-ignitions" {
  count        = var.controller_count
  content      = data.template_file.controller-configs.*.rendered[count.index]
  pretty_print = false
-  snippets     = var.controller_clc_snippets
+  snippets     = var.controller_snippets
 }

 # Controller Container Linux configs
--- a/google-cloud/container-linux/kubernetes/variables.tf
+++ b/google-cloud/container-linux/kubernetes/variables.tf
@ -64,13 +64,13 @@ variable "worker_preemptible" {
  default     = false
 }

-variable "controller_clc_snippets" {
+variable "controller_snippets" {
  type        = list(string)
  description = "Controller Container Linux Config snippets"
  default     = []
 }

-variable "worker_clc_snippets" {
+variable "worker_snippets" {
  type        = list(string)
  description = "Worker Container Linux Config snippets"
  default     = []
--- a/google-cloud/container-linux/kubernetes/workers.tf
+++ b/google-cloud/container-linux/kubernetes/workers.tf
@ -17,7 +17,7 @@ module "workers" {
  ssh_authorized_key    = var.ssh_authorized_key
  service_cidr          = var.service_cidr
  cluster_domain_suffix = var.cluster_domain_suffix
-  clc_snippets          = var.worker_clc_snippets
+  snippets              = var.worker_snippets
  node_labels           = var.worker_node_labels
 }

--- a/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/google-cloud/container-linux/kubernetes/workers/cl/worker.yaml
@ -63,7 +63,7 @@ systemd:
          --mount volume=var-log,target=/var/log \
          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          docker://quay.io/poseidon/kubelet:v1.18.0 -- \
+          docker://quay.io/poseidon/kubelet:v1.18.1 -- \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -125,7 +125,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.0 \
+            docker://quay.io/poseidon/kubelet:v1.18.1 \
            --net=host \
            --dns=host \
            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/google-cloud/container-linux/kubernetes/workers/variables.tf
+++ b/google-cloud/container-linux/kubernetes/workers/variables.tf
@ -52,7 +52,7 @@ variable "preemptible" {
  default     = false
 }

-variable "clc_snippets" {
+variable "snippets" {
  type        = list(string)
  description = "Container Linux Config snippets"
  default     = []
--- a/google-cloud/container-linux/kubernetes/workers/workers.tf
+++ b/google-cloud/container-linux/kubernetes/workers/workers.tf
@ -73,7 +73,7 @@ resource "google_compute_instance_template" "worker" {
 data "ct_config" "worker-ignition" {
  content      = data.template_file.worker-config.rendered
  pretty_print = false
-  snippets     = var.clc_snippets
+  snippets     = var.snippets
 }

 # Worker Container Linux config
--- a/google-cloud/fedora-coreos/kubernetes/README.md
+++ b/google-cloud/fedora-coreos/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.0 (upstream)
+* Kubernetes v1.18.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
+++ b/google-cloud/fedora-coreos/kubernetes/bootstrap.tf
@ -1,6 +1,6 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=cb170f802d09dcbe88b050257bc676e25d3c4282"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=1ad53d3b1c1ad75a4ed27f124f772fc5dc025245"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
--- a/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/fcc/controller.yaml
@ -28,7 +28,7 @@ systemd:
          --network host \
          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.5
+          quay.io/coreos/etcd:v3.4.7
        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
@ -79,7 +79,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.0 \
+          quay.io/poseidon/kubelet:v1.18.1 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -123,7 +123,7 @@ systemd:
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.0
+            quay.io/poseidon/kubelet:v1.18.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
@ -156,9 +156,9 @@ storage:
          sudo mkdir -p /opt/bootstrap/assets
          sudo mv manifests /opt/bootstrap/assets/manifests
          sudo mkdir -p /opt/bootstrap/assets/manifests/crds
-          sudo mv manifests-networking/crd*.yaml /opt/bootstrap/assets/manifests/crds
+          sudo mv manifests-networking/{crd,cluster}*.yaml /opt/bootstrap/assets/manifests/crds 2>/dev/null || true
          sudo mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls
+          rm -rf assets auth static-manifests tls manifests-networking
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
--- a/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
+++ b/google-cloud/fedora-coreos/kubernetes/workers/fcc/worker.yaml
@ -49,7 +49,7 @@ systemd:
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
-          quay.io/poseidon/kubelet:v1.18.0 \
+          quay.io/poseidon/kubelet:v1.18.1 \
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
@ -87,7 +87,7 @@ systemd:
        Type=oneshot
        RemainAfterExit=true
        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.0 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
+        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.1 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
        [Install]
        WantedBy=multi-user.target
 storage:
--- a/google-cloud/fedora-coreos/kubernetes/workers/workers.tf
+++ b/google-cloud/fedora-coreos/kubernetes/workers/workers.tf
@ -71,9 +71,9 @@ resource "google_compute_instance_template" "worker" {

 # Worker Ignition config
 data "ct_config" "worker-ignition" {
-  content      = data.template_file.worker-config.rendered
-  strict       = true
-  snippets     = var.snippets
+  content  = data.template_file.worker-config.rendered
+  strict   = true
+  snippets = var.snippets
 }

 # Worker Fedora CoreOS config
Author	SHA1	Message	Date
Dalton Hubble	1420700bc0	Update CHANGES for v1.18.1 release * Change order of modules in the README	2020-04-11 13:23:49 -07:00
Dalton Hubble	80538e2953	Add support for Fedora CoreOS on DigitalOcean * Add `digital-ocean/fedora-coreos/kubernetes` module * DigitalOcean custom uploaded images do not permit droplet IPv6 networking	2020-04-09 23:55:29 -07:00
Dalton Hubble	73af2f3b7c	Update Kubernetes from v1.18.0 to v1.18.1 * https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1181	2020-04-08 19:41:48 -07:00
Dalton Hubble	17ea547723	Update etcd from v3.4.5 to v3.4.7 * https://github.com/etcd-io/etcd/releases/tag/v3.4.7 * https://github.com/etcd-io/etcd/releases/tag/v3.4.6	2020-04-06 21:09:25 -07:00
Dalton Hubble	2b5dfece93	Update Grafana from v6.7.1 to v6.7.2 * https://github.com/grafana/grafana/releases/tag/v6.7.2	2020-04-04 13:13:19 -07:00
Dalton Hubble	d47d40b517	Refresh Prometheus rules/alerts and Grafana dashboards * Refresh upstream Prometheus rules and alerts and Grafana dashboards * All Loki recording rules for convenience	2020-03-31 21:53:01 -07:00
Dalton Hubble	3c1be7b0e0	Fix terraform fmt	2020-03-31 21:42:51 -07:00
Dalton Hubble	bbbaf949f9	Fix UDP outbound and clock sync timeouts on Azure workers * Add "lb" outbound rule for worker TCP _and_ UDP traffic * Fix Azure worker nodes clock synchronization being inactive due to timeouts reaching the CoreOS / Flatcar NTP pool * Fix Azure worker nodes not providing outbount UDP connectivity Background: Azure provides VMs outbound connectivity either by having a public IP or via an SNAT masquerade feature bundled with their virtual load balancing abstraction (in contrast with, say, a NAT gateway). Azure worker nodes have only a private IP, but are associated with the cluster load balancer's backend pool and ingress frontend IP. Outbound traffic uses SNAT with this frontend IP. A subtle detail with Azure SNAT seems to be that since both inbound lb_rule's are TCP only, outbound UDP traffic isn't SNAT'd (highlights the reasons Azure shouldn't have conflated inbound load balancing with outbound SNAT concepts). However, adding a separate outbound rule and disabling outbound SNAT on our ingress lb_rule's we can tell Azure to continue load balancing as before, and support outbound SNAT for worker traffic of both the TCP and UDP protocol. Fixes clock synchronization timeouts: ``` systemd-timesyncd[786]: Timed out waiting for reply from 45.79.36.123:123 (3.flatcar.pool.ntp.org) ``` Azure controller nodes have their own public IP, so controllers (and etcd) nodes have not had clock synchronization or outbound UDP issues	2020-03-31 21:00:16 -07:00
Dalton Hubble	135c6182b8	Update flannel from v0.11.0 to v0.12.0 * https://github.com/coreos/flannel/releases/tag/v0.12.0	2020-03-31 18:31:59 -07:00
Dalton Hubble	c53dc66d4a	Rename Container Linux snippets variable for consistency * Rename controller_clc_snippets to controller_snippets (cloud platforms) * Rename worker_clc_snippets to worker_snippets (cloud platforms) * Rename clc_snippets to snippets (bare-metal)	2020-03-31 18:25:51 -07:00
Dalton Hubble	9960972726	Fix bootstrap regression when networking="flannel" * Fix bootstrap error for missing `manifests-networking/crdyaml` when `networking = "flannel"` Cleanup manifest-networking directory left during bootstrap * Regressed in v1.18.0 changes for Calico https://github.com/poseidon/typhoon/pull/675	2020-03-31 18:21:59 -07:00
Dalton Hubble	bac5acb3bd	Change default kube-system DaemonSet tolerations * Change kube-proxy, flannel, and calico-node DaemonSet tolerations to tolerate `node.kubernetes.io/not-ready` and `node-role.kubernetes.io/master` (i.e. controllers) explicitly, rather than tolerating all taints * kube-system DaemonSets will no longer tolerate custom node taints by default. Instead, custom node taints must be enumerated to opt-in to scheduling/executing the kube-system DaemonSets * Consider setting the daemonset_tolerations variable of terraform-render-bootstrap at a later date Background: Tolerating all taints ruled out use-cases where certain nodes might legitimately need to keep kube-proxy or CNI networking disabled Related: https://github.com/poseidon/terraform-render-bootstrap/pull/179	2020-03-31 01:00:45 -07:00