Remove Fedora Atomic modules

* Typhoon for Fedora Atomic was deprecated in March 2019 * https://typhoon.psdn.io/announce/#march-27-2019
Remove providers block, unused with Terraform v0.12
2025-08-28 00:38:29 +02:00 · 2019-06-23 13:40:51 -07:00 · 2019-06-23 13:34:33 -07:00 · 2019-06-23 13:12:18 -07:00 · 2019-06-22 23:38:42 -07:00 · 2019-06-22 23:38:42 -07:00
201 changed files with 9391 additions and 11505 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@ -5,7 +5,7 @@
 ### Environment

 * Platform: aws, azure, bare-metal, google-cloud, digital-ocean
-* OS: container-linux, flatcar-linux, or fedora-atomic
+* OS: container-linux, flatcar-linux
 * Release: Typhoon version or Git SHA (reporting latest is **not** helpful)
 * Terraform: `terraform version` (reporting latest is **not** helpful)
 * Plugins: Provider plugin versions (reporting latest is **not** helpful)
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,204 @@ Notable changes between versions.

 ## Latest

+## v1.15.0
+
+* Kubernetes [v1.15.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#v1150)
+* Migrate from Terraform v0.11 to v0.12.x (**action required!**)
+  * [Migration](https://typhoon.psdn.io/topics/maintenance/#terraform-v012x) instructions for Terraform v0.12
+* Require `terraform-provider-ct` v0.3.2+ to support Terraform v0.12 (action required)
+* Update Calico from v3.7.2 to [v3.7.3](https://docs.projectcalico.org/v3.7/release-notes/)
+* Remove Fedora Atomic modules (deprecated in March) ([#501](https://github.com/poseidon/typhoon/pull/501))
+
+#### AWS
+
+* Require `terraform-provider-aws` v2.7+ to support Terraform v0.12 (action required)
+* Allow using Flatcar Linux Edge by setting `os_image` to "flatcar-edge"
+
+#### Azure
+
+* Require `terraform-provider-azurerm` v1.27+ to support Terraform v0.12 (action required)
+* Avoid unneeded rotations of Regular priority virtual machine scale sets
+  * Azure only allows `eviction_policy` to be set for Low priority VMs. Supporting Low priority VMs meant when Regular VMs were used, each `terraform apply` rolled workers, to set eviction_policy to null.
+  * Terraform v0.12 nullable variables fix the issue so plan does not produce a diff. 
+
+#### Bare-Metal
+
+* Require `terraform-provider-matchbox` v0.3.0+ to support Terraform v0.12 (action required)
+* Allow using Flatcar Linux Edge by setting `os_channel` to "flatcar-edge"
+
+#### DigitalOcean
+
+* Require `terraform-provider-digitalocean` v1.3+ to support Terraform v0.12 (action required)
+* Change the default `worker_type` from `s-1vcpu1-1gb` to `s-1vcpu-2gb`
+
+#### Google Cloud
+
+* Require `terraform-provider-google` v2.5+ to support Terraform v0.12 (action required)
+
+#### Addons
+
+* Update Grafana from v6.2.1 to v6.2.4
+* Update node-exporter from v0.18.0 to v0.18.1
+
+## v1.14.3
+
+* Kubernetes [v1.14.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#v1143)
+* Update CoreDNS from v1.3.1 to v1.5.0
+  * Add `ready` plugin to improve readinessProbe
+* Fix trailing slash in terraform-render-bootkube version ([#479](https://github.com/poseidon/typhoon/pull/479))
+* Recommend updating `terraform-provider-ct` plugin from v0.3.1 to [v0.3.2](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.3.2) ([#487](https://github.com/poseidon/typhoon/pull/487))
+
+#### AWS
+
+* Rename `worker` pool module `count` variable to `worker_count` ([#485](https://github.com/poseidon/typhoon/pull/485)) (action required)
+  * `count` will become a reserved variable name in Terraform v0.12
+
+#### Azure
+
+* Replace `azurerm_autoscale_setting` with `azurerm_monitor_autoscale_setting` ([#482](https://github.com/poseidon/typhoon/pull/482))
+* Rename `worker` pool module `count` variable to `worker_count` ([#485](https://github.com/poseidon/typhoon/pull/485)) (action required)
+  * `count` will become a reserved variable name in Terraform v0.12
+
+#### Bare-Metal
+
+* Recommend updating `terraform-provider-matchbox` plugin from v0.2.3 to [v0.3.0](https://github.com/poseidon/terraform-provider-matchbox/releases/tag/v0.3.0) ([#487](https://github.com/poseidon/typhoon/pull/487))
+
+#### Google Cloud
+
+* Rename `worker` pool module `count` variable to `worker_count` ([#485](https://github.com/poseidon/typhoon/pull/485)) (action required)
+  * `count` is a reserved variable in Terraform v0.12
+
+#### Addons
+
+* Update Prometheus from v2.9.2 to v2.10.0
+* Update Grafana from v6.1.6 to v6.2.1
+
+## v1.14.2
+ 
+* Kubernetes [v1.14.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#v1142)
+* Update etcd from v3.3.12 to [v3.3.13](https://github.com/etcd-io/etcd/releases/tag/v3.3.13)
+* Upgrade Calico from v3.6.1 to [v3.7.2](https://docs.projectcalico.org/v3.7/release-notes/)
+* Change flannel VXLAN port from 8472 (kernel default) to 4789 (IANA VXLAN)
+
+#### AWS
+
+* Only set internal VXLAN rules when `networking` is "flannel" (default: calico)
+
+#### Azure
+
+* Allow choosing Calico as the network provider (experimental) ([#472](https://github.com/poseidon/typhoon/pull/472))
+  * Add a `networking` variable accepting "flannel" (default) or "calico"
+  * Use VXLAN encapsulation since Azure doesn't support IPIP
+
+#### DigitalOcean
+
+* Allow choosing Calico as the network provider (experimental) ([#472](https://github.com/poseidon/typhoon/pull/472))
+  * Add a `networking` variable accepting "flannel" (default) or "calico"
+  * Use VXLAN encapsulation since DigitalOcean doesn't support IPIP
+* Add explicit ordering between firewall rule creation and secure copying Kubelet credentials ([#469](https://github.com/poseidon/typhoon/pull/469))
+  * Fix race scenario if copies to nodes were before rule creation, blocking cluster creation
+
+#### Addons
+
+* Update Prometheus from v2.8.1 to v2.9.2
+  * Update kube-state-metrics from v1.5.0 to v1.6.0
+* Update node-exporter from v0.17.0 to v0.18.0
+* Update Grafana from v6.1.3 to v6.1.6
+* Reduce nginx-ingress Role RBAC permissions ([#458](https://github.com/poseidon/typhoon/pull/458))
+
+## v1.14.1
+
+* Kubernetes [v1.14.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#v1141)
+
+#### Addons
+
+* Update Grafana from v6.1.1 to v6.1.3
+* Update nginx-ingress from v0.23.0 to v0.24.1
+
+## v1.14.0
+
+* Kubernetes [v1.14.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#v1140)
+* Update Calico from v3.6.0 to v3.6.1
+* Add `enable_aggregation` option for CNCF conformance ([#436](https://github.com/poseidon/typhoon/pull/436))
+  * Aggregation is disabled by default to retain our security stance
+  * Aggregation increases the security surface area. Extensions become part of the control plane and must be scrutinized carefully and trusted. Favor leaving aggregation disabled.
+
+#### AWS
+
+* Add ability to load balance TCP applications ([#443](https://github.com/poseidon/typhoon/pull/443))
+  * Output the network load balancer ARN as `nlb_id`
+  * Accept a `worker_target_groups` (ARN) list to which worker instances should be added
+
+#### Azure
+
+* Add ability to load balance TCP/UDP applications ([#447](https://github.com/poseidon/typhoon/pull/447))
+  * Output the load balancer ID as `loadbalancer_id`
+* Output `worker_security_group_name` and `worker_address_prefix` for extending firewall rules ([#447](https://github.com/poseidon/typhoon/pull/447))
+
+#### DigitalOcean
+
+* Harden internal (node-to-node) firewall rules to align with other platforms ([#444](https://github.com/poseidon/typhoon/pull/444))
+* Add ability to load balance TCP applications ([#444](https://github.com/poseidon/typhoon/pull/444))
+  * Output `controller_tag` and `worker_tag` for extending firewall rules ([#444](https://github.com/poseidon/typhoon/pull/444))
+
+#### Google Cloud
+
+* Add ability to load balance TCP/UDP applications ([#442](https://github.com/poseidon/typhoon/pull/442))
+  * Add worker instances to a target pool, output as `worker_target_pool`
+  * Health check for workers with Ingress controllers. Forward rules don't support differing internal/external ports, but some Ingress controllers support TCP/UDP proxy as a workaround 
+* Remove Haswell minimum CPU platform requirement ([#439](https://github.com/poseidon/typhoon/pull/439))
+  * Google Cloud API implements `min_cpu_platform` to mean "use exactly this CPU". Revert [#405](https://github.com/poseidon/typhoon/pull/405) added in v1.13.4.
+  * Fix error creating clusters in new regions without Haswell (e.g. europe-west2) ([#438](https://github.com/poseidon/typhoon/issues/438))
+
+#### Addons
+
+* Update Prometheus from v2.8.0 to v2.8.1
+* Update Grafana from v6.0.2 to [v6.1.1](http://docs.grafana.org/guides/whats-new-in-v6-1/)
+  * Add dashboard for pods in a workload (deployment/daemonset/statefulset) ([#446](https://github.com/poseidon/typhoon/pull/446))
+  * Add dashboard for workloads by namespace
+
+## v1.13.5
+
+* Kubernetes [v1.13.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1135)
+* Resolve in-addr.arpa reverse DNS lookups (PTR) for pod IPv4 addresses ([#415](https://github.com/poseidon/typhoon/pull/415))
+  * Reverse DNS lookups for service IPv4 addresses unchanged
+* Upgrade Calico from v3.5.2 to [v3.6.0](https://docs.projectcalico.org/v3.6/release-notes/) ([#430](https://github.com/poseidon/typhoon/pull/430))
+  * Change pod IPAM from `host-local` to `calico-ipam`. `pod_cidr` is still divided into `/24` subnets per node, but managed as `ippools` and `ipamblocks`
+* Recommend updating [terraform-provider-ct](https://github.com/poseidon/terraform-provider-ct) from v0.3.0 to [v0.3.1](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.3.1) ([#434](https://github.com/poseidon/typhoon/pull/434))
+* Announce: Fedora Atomic modules will be not be updated beyond Kubernetes v1.13.x ([#437](https://github.com/poseidon/typhoon/pull/437))
+  * Thank you Project Atomic team and users, please see the deprecation [notice](https://typhoon.psdn.io/announce/#march-27-2019)
+
+#### AWS
+
+* Support `terraform-provider-aws` v2.0+ ([#419](https://github.com/poseidon/typhoon/pull/419))
+
+#### Bare-Metal
+
+* Change the default iPXE kernel and initrd download protocol from HTTP to HTTPS ([#420](https://github.com/poseidon/typhoon/pull/420))
+  * Require an iPXE-enabled network boot environment with support for TLS downloads. PXE clients must chainload to iPXE firmware compiled with `DOWNLOAD_PROTO_HTTPS` [enabled](https://ipxe.org/crypto). (**action required**)
+  * Only affects Container Linux and Flatcar Linux install profiles that pull public images (default)
+  * Add `download_protocol` variable. Recognizing boot firmware TLS support is difficult in some environments, set the protocol to "http" for the old behavior (discouraged)
+
+#### DigitalOcean
+
+* Fix kubelet hostname-override to set node metadata InternalIP correctly ([#424](https://github.com/poseidon/typhoon/issues/424))
+  * Uniquely, DigitalOcean does not resolve hostnames to instance private IPs. Kubelet auto-detect mechanisms require the internal IP be set directly.
+  * Regressed in v1.12.3 ([#337](https://github.com/poseidon/typhoon/pull/337)) which aimed to provide friendly hostname-based node names on DigitalOcean
+
+#### Addons
+
+* Update Prometheus from v2.7.1 to [v2.8.0](https://github.com/prometheus/prometheus/releases/tag/v2.8.0)
+  * Refresh rules based on upstreams ([#426](https://github.com/poseidon/typhoon/pull/426))
+  * Define NetworkPolicy to allow only traffic from the Grafana addon
+* Update Grafana from v6.0.0 to v6.0.2
+  * Add liveness and readiness probes
+  * Refresh dashboards and organize to stay below ConfigMap size limit ([#426](https://github.com/poseidon/typhoon/pull/426))
+* Remove heapster manifests from addons ([#427](https://github.com/poseidon/typhoon/pull/427))
+  * Heapster addon powers `kubectl top` (in early Kubernetes, running the addon was expected). Today, there are better monitoring options.
+  * `kubectl top` reliance on a non-core extension means its not in-scope for minimal Kubernetes
+  * Look to prior releases if you still wish to apply heapster
+
 ## v1.13.4

 * Kubernetes [v1.13.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#v1134)
@ -15,7 +213,7 @@ Notable changes between versions.

 #### Bare-Metal

-* Recommend updating [terraform-provider-matchbox](https://github.com/coreos/terraform-provider-matchbox) plugin from v0.2.2 to [v0.2.3](https://github.com/coreos/terraform-provider-matchbox/releases/tag/v0.2.3) ([#402](https://github.com/poseidon/typhoon/pull/402))
+* Recommend updating [terraform-provider-matchbox](https://github.com/poseidon/terraform-provider-matchbox) plugin from v0.2.2 to [v0.2.3](https://github.com/poseidon/terraform-provider-matchbox/releases/tag/v0.2.3) ([#402](https://github.com/poseidon/typhoon/pull/402))
 * Improve docs on using Ubiquiti EdgeOS with bare-metal clusters ([#413](https://github.com/poseidon/typhoon/pull/413))

 #### Google Cloud
@ -527,15 +725,15 @@ Notable changes between versions.

 #### AWS

-* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
+* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.2.1) (action required!)

 #### Digital Ocean

-* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
+* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.2.1) (action required!)

 #### Google Cloud

-* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
+* [Require](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.2.1) (action required!)
 * Relax `os_image` to optional. Default to "coreos-stable".

 #### Addons
@ -555,7 +753,7 @@ Notable changes between versions.
 * Upgrade etcd from v3.2.15 to v3.3.2
 * Update Calico from v3.0.2 to v3.0.3
 * Use kubernetes-incubator/bootkube v0.11.0
-* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/coreos/terraform-provider-ct/releases/tag/v0.2.1) (action recommended)
+* [Recommend](https://typhoon.psdn.io/topics/maintenance/#terraform-provider-ct-v021) updating `terraform-provider-ct` plugin from v0.2.0 to [v0.2.1](https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.2.1) (action recommended)

 #### AWS

--- a/README.md
+++ b/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
@ -19,25 +19,16 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Modules

-Typhoon provides a Terraform Module for each supported operating system and platform. Container Linux is a mature and reliable choice. Also, Kinvolk's Flatcar Linux fork is selectable on AWS and bare-metal.
+Typhoon provides a Terraform Module for each supported operating system and platform.

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Container Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
+| AWS           | Container Linux / Flatcar Linux  | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
 | Azure         | Container Linux  | [azure/container-linux/kubernetes](cl/azure.md) | alpha |
-| Bare-Metal    | Container Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
+| Bare-Metal    | Container Linux / Flatcar Linux  | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
 | Digital Ocean | Container Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
 | Google Cloud  | Container Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | stable |

-Fedora Atomic support is alpha and will evolve as Fedora Atomic is replaced by Fedora CoreOS.
-
-| Platform      | Operating System | Terraform Module | Status |
-|---------------|------------------|------------------|--------|
-| AWS           | Fedora Atomic    | [aws/fedora-atomic/kubernetes](aws/fedora-atomic/kubernetes) | alpha |
-| Bare-Metal    | Fedora Atomic    | [bare-metal/fedora-atomic/kubernetes](bare-metal/fedora-atomic/kubernetes) | alpha |
-| Digital Ocean | Fedora Atomic    | [digital-ocean/fedora-atomic/kubernetes](digital-ocean/fedora-atomic/kubernetes) | alpha |
-| Google Cloud  | Fedora Atomic    | [google-cloud/fedora-atomic/kubernetes](google-cloud/fedora-atomic/kubernetes) | alpha |
-
 ## Documentation

 * [Docs](https://typhoon.psdn.io)
@ -50,15 +41,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "google-cloud-yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.13.4"
-  
-  providers = {
-    google   = "google.default"
-    local    = "local.default"
-    null     = "null.default"
-    template = "template.default"
-    tls      = "tls.default"
-  }
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/container-linux/kubernetes?ref=v1.15.0"

  # Google Cloud
  cluster_name  = "yavin"
@ -72,6 +55,7 @@ module "google-cloud-yavin" {
  
  # optional
  worker_count = 2
+  worker_preemptible = true
 }
 ```

@ -91,9 +75,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.secrets/clusters/yavin/auth/kubeconfig
 $ kubectl get nodes
 NAME                                       ROLES              STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.13.4
-yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.13.4
-yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.13.4
+yavin-controller-0.c.example-com.internal  controller,master  Ready   6m   v1.15.0
+yavin-worker-jrbf.c.example-com.internal   node               Ready   5m   v1.15.0
+yavin-worker-mzdm.c.example-com.internal   node               Ready   5m   v1.15.0
 ```

 List the pods.
@ -145,3 +129,5 @@ Typhoon clusters will contain only [free](https://www.debian.org/intro/free) com
 ## Donations

 Typhoon does not accept money donations. Instead, we encourage you to donate to one of [these organizations](https://github.com/poseidon/typhoon/wiki/Donations) to show your appreciation.
+
+* [DigitalOcean](https://www.digitalocean.com/) kindly provides credits to support Typhoon test clusters.
--- a/addons/cluo/update-agent.yaml
+++ b/addons/cluo/update-agent.yaml
@ -18,34 +18,41 @@ spec:
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
-      containers:
-      - name: update-agent
-        image: quay.io/coreos/container-linux-update-operator:v0.7.0
-        command:
-        - "/bin/update-agent"
-        volumeMounts:
-          - mountPath: /var/run/dbus
-            name: var-run-dbus
-          - mountPath: /etc/coreos
-            name: etc-coreos
-          - mountPath: /usr/share/coreos
-            name: usr-share-coreos
-          - mountPath: /etc/os-release
-            name: etc-os-release
-        env:
-        # read by update-agent as the node name to manage reboots for
-        - name: UPDATE_AGENT_NODE
-          valueFrom:
-            fieldRef:
-              fieldPath: spec.nodeName
-        - name: POD_NAMESPACE
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.namespace
      tolerations:
-      - key: node-role.kubernetes.io/master
-        operator: Exists
-        effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: update-agent
+          image: quay.io/coreos/container-linux-update-operator:v0.7.0
+          command:
+            - "/bin/update-agent"
+          env:
+            # read by update-agent as the node name to manage reboots for
+            - name: UPDATE_AGENT_NODE
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: 10m
+              memory: 20Mi
+            limits:
+              cpu: 20m
+              memory: 40Mi
+          volumeMounts:
+            - mountPath: /var/run/dbus
+              name: var-run-dbus
+            - mountPath: /etc/coreos
+              name: etc-coreos
+            - mountPath: /usr/share/coreos
+              name: usr-share-coreos
+            - mountPath: /etc/os-release
+              name: etc-os-release
      volumes:
      - name: var-run-dbus
        hostPath:
--- a/addons/cluo/update-operator.yaml
+++ b/addons/cluo/update-operator.yaml
@ -15,17 +15,25 @@ spec:
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
-      containers:
-      - name: update-operator
-        image: quay.io/coreos/container-linux-update-operator:v0.7.0
-        command:
-        - "/bin/update-operator"
-        env:
-        - name: POD_NAMESPACE
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.namespace
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
+      containers:
+        - name: update-operator
+          image: quay.io/coreos/container-linux-update-operator:v0.7.0
+          command:
+            - "/bin/update-operator"
+          env:
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: 10m
+              memory: 20Mi
+            limits:
+              cpu: 20m
+              memory: 40Mi
+
--- a/addons/grafana/dashboards-etcd.yaml
+++ b/addons/grafana/dashboards-etcd.yaml
@ -0,0 +1,1230 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboards-etcd
+  namespace: monitoring
+data:
+  etcd.json: |-
+    {
+      "annotations": {
+        "list": [
+
+        ]
+      },
+      "description": "etcd sample Grafana dashboard with Prometheus",
+      "editable": true,
+      "gnetId": null,
+      "hideControls": false,
+      "id": 6,
+      "links": [
+
+      ],
+      "refresh": false,
+      "rows": [
+        {
+          "collapse": false,
+          "editable": true,
+          "height": "250px",
+          "panels": [
+            {
+              "cacheTimeout": null,
+              "colorBackground": false,
+              "colorValue": false,
+              "colors": [
+                "rgba(245, 54, 54, 0.9)",
+                "rgba(237, 129, 40, 0.89)",
+                "rgba(50, 172, 45, 0.97)"
+              ],
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "format": "none",
+              "gauge": {
+                "maxValue": 100,
+                "minValue": 0,
+                "show": false,
+                "thresholdLabels": false,
+                "thresholdMarkers": true
+              },
+              "id": 28,
+              "interval": null,
+              "isNew": true,
+              "links": [
+
+              ],
+              "mappingType": 1,
+              "mappingTypes": [
+                {
+                  "name": "value to text",
+                  "value": 1
+                },
+                {
+                  "name": "range to text",
+                  "value": 2
+                }
+              ],
+              "maxDataPoints": 100,
+              "nullPointMode": "connected",
+              "nullText": null,
+              "postfix": "",
+              "postfixFontSize": "50%",
+              "prefix": "",
+              "prefixFontSize": "50%",
+              "rangeMaps": [
+                {
+                  "from": "null",
+                  "text": "N/A",
+                  "to": "null"
+                }
+              ],
+              "span": 3,
+              "sparkline": {
+                "fillColor": "rgba(31, 118, 189, 0.18)",
+                "full": false,
+                "lineColor": "rgb(31, 120, 193)",
+                "show": false
+              },
+              "targets": [
+                {
+                  "expr": "sum(etcd_server_has_leader{job=\"$cluster\"})",
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "metric": "etcd_server_has_leader",
+                  "refId": "A",
+                  "step": 20
+                }
+              ],
+              "thresholds": "",
+              "title": "Up",
+              "type": "singlestat",
+              "valueFontSize": "200%",
+              "valueMaps": [
+                {
+                  "op": "=",
+                  "text": "N/A",
+                  "value": "null"
+                }
+              ],
+              "valueName": "avg"
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "id": 23,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 5,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(rate(grpc_server_started_total{job=\"$cluster\",grpc_type=\"unary\"}[5m]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "RPC Rate",
+                  "metric": "grpc_server_started_total",
+                  "refId": "A",
+                  "step": 2
+                },
+                {
+                  "expr": "sum(rate(grpc_server_handled_total{job=\"$cluster\",grpc_type=\"unary\",grpc_code!=\"OK\"}[5m]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "RPC Failed Rate",
+                  "metric": "grpc_server_handled_total",
+                  "refId": "B",
+                  "step": 2
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "RPC Rate",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "ops",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "id": 41,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 4,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(grpc_server_started_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Watch\",grpc_type=\"bidi_stream\"}) - sum(grpc_server_handled_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Watch\",grpc_type=\"bidi_stream\"})",
+                  "intervalFactor": 2,
+                  "legendFormat": "Watch Streams",
+                  "metric": "grpc_server_handled_total",
+                  "refId": "A",
+                  "step": 4
+                },
+                {
+                  "expr": "sum(grpc_server_started_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Lease\",grpc_type=\"bidi_stream\"}) - sum(grpc_server_handled_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Lease\",grpc_type=\"bidi_stream\"})",
+                  "intervalFactor": 2,
+                  "legendFormat": "Lease Streams",
+                  "metric": "grpc_server_handled_total",
+                  "refId": "B",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Active Streams",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": "",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "showTitle": false,
+          "title": "Row"
+        },
+        {
+          "collapse": false,
+          "editable": true,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "decimals": null,
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "grid": {
+
+              },
+              "id": 1,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 4,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "etcd_debugging_mvcc_db_total_size_in_bytes{job=\"$cluster\"}",
+                  "hide": false,
+                  "interval": "",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} DB Size",
+                  "metric": "",
+                  "refId": "A",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "DB Size",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "cumulative"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "grid": {
+
+              },
+              "id": 3,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 1,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 4,
+              "stack": false,
+              "steppedLine": true,
+              "targets": [
+                {
+                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
+                  "hide": false,
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} WAL fsync",
+                  "metric": "etcd_disk_wal_fsync_duration_seconds_bucket",
+                  "refId": "A",
+                  "step": 4
+                },
+                {
+                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} DB fsync",
+                  "metric": "etcd_disk_backend_commit_duration_seconds_bucket",
+                  "refId": "B",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Disk Sync Duration",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "cumulative"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "s",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "id": 29,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 4,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "process_resident_memory_bytes{job=\"$cluster\"}",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Resident Memory",
+                  "metric": "process_resident_memory_bytes",
+                  "refId": "A",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "title": "New row"
+        },
+        {
+          "collapse": false,
+          "editable": true,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 5,
+              "id": 22,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 3,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "rate(etcd_network_client_grpc_received_bytes_total{job=\"$cluster\"}[5m])",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Client Traffic In",
+                  "metric": "etcd_network_client_grpc_received_bytes_total",
+                  "refId": "A",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Client Traffic In",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 5,
+              "id": 21,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 3,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "rate(etcd_network_client_grpc_sent_bytes_total{job=\"$cluster\"}[5m])",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Client Traffic Out",
+                  "metric": "etcd_network_client_grpc_sent_bytes_total",
+                  "refId": "A",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Client Traffic Out",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "id": 20,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 3,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(rate(etcd_network_peer_received_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Peer Traffic In",
+                  "metric": "etcd_network_peer_received_bytes_total",
+                  "refId": "A",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Peer Traffic In",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "decimals": null,
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "grid": {
+
+              },
+              "id": 16,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 3,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(rate(etcd_network_peer_sent_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
+                  "hide": false,
+                  "interval": "",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Peer Traffic Out",
+                  "metric": "etcd_network_peer_sent_bytes_total",
+                  "refId": "A",
+                  "step": 4
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Peer Traffic Out",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "cumulative"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "title": "New row"
+        },
+        {
+          "collapse": false,
+          "editable": true,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "id": 40,
+              "isNew": true,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 6,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(rate(etcd_server_proposals_failed_total{job=\"$cluster\"}[5m]))",
+                  "intervalFactor": 2,
+                  "legendFormat": "Proposal Failure Rate",
+                  "metric": "etcd_server_proposals_failed_total",
+                  "refId": "A",
+                  "step": 2
+                },
+                {
+                  "expr": "sum(etcd_server_proposals_pending{job=\"$cluster\"})",
+                  "intervalFactor": 2,
+                  "legendFormat": "Proposal Pending Total",
+                  "metric": "etcd_server_proposals_pending",
+                  "refId": "B",
+                  "step": 2
+                },
+                {
+                  "expr": "sum(rate(etcd_server_proposals_committed_total{job=\"$cluster\"}[5m]))",
+                  "intervalFactor": 2,
+                  "legendFormat": "Proposal Commit Rate",
+                  "metric": "etcd_server_proposals_committed_total",
+                  "refId": "C",
+                  "step": 2
+                },
+                {
+                  "expr": "sum(rate(etcd_server_proposals_applied_total{job=\"$cluster\"}[5m]))",
+                  "intervalFactor": 2,
+                  "legendFormat": "Proposal Apply Rate",
+                  "refId": "D",
+                  "step": 2
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Raft Proposals",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": "",
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "datasource": "$datasource",
+              "decimals": 0,
+              "editable": true,
+              "error": false,
+              "fill": 0,
+              "id": 19,
+              "isNew": true,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "span": 6,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "changes(etcd_server_leader_changes_seen_total{job=\"$cluster\"}[1d])",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Total Leader Elections Per Day",
+                  "metric": "etcd_server_leader_changes_seen_total",
+                  "refId": "A",
+                  "step": 2
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Total Leader Elections Per Day",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "title": "New row"
+        }
+      ],
+      "schemaVersion": 13,
+      "sharedCrosshair": false,
+      "style": "dark",
+      "tags": [
+
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "Prometheus",
+              "value": "Prometheus"
+            },
+            "hide": 0,
+            "label": null,
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(etcd_server_has_leader, job)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-15m",
+        "to": "now"
+      },
+      "timepicker": {
+        "now": true,
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "browser",
+      "title": "etcd",
+      "version": 215
+    }
--- a/addons/grafana/dashboards-k8s-resources.yaml
+++ b/addons/grafana/dashboards-k8s-resources.yaml
@ -0,0 +1,5252 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboards-k8s-resources
+  namespace: monitoring
+data:
+  k8s-resources-cluster.json: |-
+    {
+      "annotations": {
+        "list": [
+
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "links": [
+
+      ],
+      "refresh": "10s",
+      "rows": [
+        {
+          "collapse": false,
+          "height": "100px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "format": "percentunit",
+              "id": 1,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 2,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\", cluster=\"$cluster\"}[1m]))",
+                  "format": "time_series",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "70,80",
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Utilisation",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "singlestat",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "format": "percentunit",
+              "id": 2,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 2,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) / sum(node:node_num_cpu:sum{cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "70,80",
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Requests Commitment",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "singlestat",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "format": "percentunit",
+              "id": 3,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 2,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) / sum(node:node_num_cpu:sum{cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "70,80",
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Limits Commitment",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "singlestat",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "format": "percentunit",
+              "id": 4,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 2,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "1 - sum(:node_memory_MemFreeCachedBuffers_bytes:sum{cluster=\"$cluster\"}) / sum(:node_memory_MemTotal_bytes:sum{cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "70,80",
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Utilisation",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "singlestat",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "format": "percentunit",
+              "id": 5,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 2,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) / sum(:node_memory_MemTotal_bytes:sum{cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "70,80",
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Requests Commitment",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "singlestat",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "format": "percentunit",
+              "id": 6,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 2,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) / sum(:node_memory_MemTotal_bytes:sum{cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "70,80",
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Limits Commitment",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "singlestat",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": false,
+          "title": "Headlines",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 7,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{namespace}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 8,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Pods",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 0,
+                  "link": true,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Workloads",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 0,
+                  "link": true,
+                  "linkTooltip": "Drill down to workloads",
+                  "linkUrl": "/d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "CPU Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #G",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Namespace",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "pattern": "namespace",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "count(avg(mixin_pod_workload{cluster=\"$cluster\"}) by (workload, namespace)) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "G",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Quota",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 9,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container_name!=\"\"}) by (namespace)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{namespace}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Usage (w/o cache)",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 10,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Pods",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 0,
+                  "link": true,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Workloads",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 0,
+                  "link": true,
+                  "linkTooltip": "Drill down to workloads",
+                  "linkUrl": "/d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Memory Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #G",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Namespace",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "pattern": "namespace",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "count(avg(mixin_pod_workload{cluster=\"$cluster\"}) by (workload, namespace)) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container_name!=\"\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container_name!=\"\"}) by (namespace) / sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container_name!=\"\"}) by (namespace) / sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "G",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Requests by Namespace",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Requests",
+          "titleSize": "h6"
+        }
+      ],
+      "schemaVersion": 14,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "Prometheus",
+              "value": "Prometheus"
+            },
+            "hide": 0,
+            "label": null,
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(node_cpu_seconds_total, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "",
+      "title": "Kubernetes / Compute Resources / Cluster",
+      "uid": "efa86fd1d0c121a26444b636a3f509a8",
+      "version": 0
+    }
+  k8s-resources-namespace.json: |-
+    {
+      "annotations": {
+        "list": [
+
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "links": [
+
+      ],
+      "refresh": "10s",
+      "rows": [
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 1,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod_name)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod_name}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 2,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "CPU Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "CPU Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Pod",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "pattern": "pod",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Quota",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 3,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"}) by (pod_name)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod_name}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Usage (w/o cache)",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 4,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Memory Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Usage (RSS)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Usage (Cache)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #G",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Usage (Swap",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #H",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Pod",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "pattern": "pod",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum(label_replace(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"}) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "G",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "H",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Quota",
+          "titleSize": "h6"
+        }
+      ],
+      "schemaVersion": 14,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "Prometheus",
+              "value": "Prometheus"
+            },
+            "hide": 0,
+            "label": null,
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "namespace",
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "",
+      "title": "Kubernetes / Compute Resources / Namespace (Pods)",
+      "uid": "85a562078cdf77779eaa1add43ccec1e",
+      "version": 0
+    }
+  k8s-resources-pod.json: |-
+    {
+      "annotations": {
+        "list": [
+
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "links": [
+
+      ],
+      "refresh": "10s",
+      "rows": [
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 1,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", cluster=\"$cluster\"}) by (container_name)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container_name}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 2,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "CPU Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "CPU Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Container",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "container",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Quota",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 3,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}) by (container_name)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container_name}} (RSS)",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}) by (container_name)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container_name}} (Cache)",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}) by (container_name)",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container_name}} (Swap)",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 4,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Memory Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Usage (RSS)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Usage (Cache)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #G",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Usage (Swap",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #H",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Container",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "container",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum(label_replace(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name != \"\", container_name != \"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name != \"\", container_name != \"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "G",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(label_replace(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name != \"\", container_name != \"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "H",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Quota",
+          "titleSize": "h6"
+        }
+      ],
+      "schemaVersion": 14,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "Prometheus",
+              "value": "Prometheus"
+            },
+            "hide": 0,
+            "label": null,
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "namespace",
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "pod",
+            "multi": false,
+            "name": "pod",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=\"$namespace\"}, pod)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "",
+      "title": "Kubernetes / Compute Resources / Pod",
+      "uid": "6581e46e4e5c7ba40a07646395ef7b23",
+      "version": 0
+    }
+  k8s-resources-workload.json: |-
+    {
+      "annotations": {
+        "list": [
+
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "links": [
+
+      ],
+      "refresh": "10s",
+      "rows": [
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 1,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 2,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "CPU Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "CPU Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Pod",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "pattern": "pod",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Quota",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 3,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n  ) by (pod)\n",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 4,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Memory Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Pod",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "pattern": "pod",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n  ) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n  ) by (pod)\n/sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n  ) by (pod)\n/sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Quota",
+          "titleSize": "h6"
+        }
+      ],
+      "schemaVersion": 14,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "Prometheus",
+              "value": "Prometheus"
+            },
+            "hide": 0,
+            "label": null,
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "namespace",
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "workload",
+            "multi": false,
+            "name": "workload",
+            "options": [
+
+            ],
+            "query": "label_values(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}, workload)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "type",
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\"}, workload_type)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "",
+      "title": "Kubernetes / Compute Resources / Workload",
+      "uid": "a164a7f0339f99e89cea5cb47e9be617",
+      "version": 0
+    }
+  k8s-resources-workloads-namespace.json: |-
+    {
+      "annotations": {
+        "list": [
+
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "links": [
+
+      ],
+      "refresh": "10s",
+      "rows": [
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 1,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{workload}} - {{workload_type}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 2,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Running Pods",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 0,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "CPU Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "CPU Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Workload",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
+                  "pattern": "workload",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Workload Type",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "workload_type",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}) by (workload, workload_type)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "CPU Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "CPU Quota",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 3,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n  ) by (workload, workload_type)\n",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{workload}} - {{workload_type}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Usage",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "bytes",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Usage",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 4,
+              "legend": {
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "Running Pods",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 0,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Memory Usage",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Requests %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Memory Limits",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "bytes"
+                },
+                {
+                  "alias": "Memory Limits %",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "percentunit"
+                },
+                {
+                  "alias": "Workload",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
+                  "pattern": "workload",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Workload Type",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "workload_type",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}) by (workload, workload_type)",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n  ) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n  ) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum(\n  label_replace(\n    container_memory_usage_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container_name!=\"\"},\n    \"pod\", \"$1\", \"pod_name\", \"(.*)\"\n  ) * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n  ) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod) group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}\n) by (workload, workload_type)\n",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Memory Quota",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Memory Quota",
+          "titleSize": "h6"
+        }
+      ],
+      "schemaVersion": 14,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "Prometheus",
+              "value": "Prometheus"
+            },
+            "hide": 0,
+            "label": null,
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": "namespace",
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "",
+      "title": "Kubernetes / Compute Resources / Namespace (Workloads)",
+      "uid": "a87fb0d919ec0ea5f6543124e16c42a5",
+      "version": 0
+    }
--- a/addons/grafana/dashboards-k8s.yaml
+++ b/addons/grafana/dashboards-k8s.yaml
@ -1,1233 +1,9 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: grafana-dashboards
+  name: grafana-dashboards-k8s
  namespace: monitoring
 data:
-  etcd.json: |-
-    {
-      "annotations": {
-        "list": [
-
-        ]
-      },
-      "description": "etcd sample Grafana dashboard with Prometheus",
-      "editable": true,
-      "gnetId": null,
-      "hideControls": false,
-      "id": 6,
-      "links": [
-
-      ],
-      "refresh": false,
-      "rows": [
-        {
-          "collapse": false,
-          "editable": true,
-          "height": "250px",
-          "panels": [
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "rgba(245, 54, 54, 0.9)",
-                "rgba(237, 129, 40, 0.89)",
-                "rgba(50, 172, 45, 0.97)"
-              ],
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "id": 28,
-              "interval": null,
-              "isNew": true,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 3,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "targets": [
-                {
-                  "expr": "sum(etcd_server_has_leader{job=\"$cluster\"})",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "metric": "etcd_server_has_leader",
-                  "refId": "A",
-                  "step": 20
-                }
-              ],
-              "thresholds": "",
-              "title": "Up",
-              "type": "singlestat",
-              "valueFontSize": "200%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "avg"
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "id": 23,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 5,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(grpc_server_started_total{job=\"$cluster\",grpc_type=\"unary\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "RPC Rate",
-                  "metric": "grpc_server_started_total",
-                  "refId": "A",
-                  "step": 2
-                },
-                {
-                  "expr": "sum(rate(grpc_server_handled_total{job=\"$cluster\",grpc_type=\"unary\",grpc_code!=\"OK\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "RPC Failed Rate",
-                  "metric": "grpc_server_handled_total",
-                  "refId": "B",
-                  "step": 2
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "RPC Rate",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "id": 41,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 4,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(grpc_server_started_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Watch\",grpc_type=\"bidi_stream\"}) - sum(grpc_server_handled_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Watch\",grpc_type=\"bidi_stream\"})",
-                  "intervalFactor": 2,
-                  "legendFormat": "Watch Streams",
-                  "metric": "grpc_server_handled_total",
-                  "refId": "A",
-                  "step": 4
-                },
-                {
-                  "expr": "sum(grpc_server_started_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Lease\",grpc_type=\"bidi_stream\"}) - sum(grpc_server_handled_total{job=\"$cluster\",grpc_service=\"etcdserverpb.Lease\",grpc_type=\"bidi_stream\"})",
-                  "intervalFactor": 2,
-                  "legendFormat": "Lease Streams",
-                  "metric": "grpc_server_handled_total",
-                  "refId": "B",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Active Streams",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": "",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "showTitle": false,
-          "title": "Row"
-        },
-        {
-          "collapse": false,
-          "editable": true,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "decimals": null,
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "grid": {
-
-              },
-              "id": 1,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "etcd_debugging_mvcc_db_total_size_in_bytes{job=\"$cluster\"}",
-                  "hide": false,
-                  "interval": "",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} DB Size",
-                  "metric": "",
-                  "refId": "A",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "DB Size",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "cumulative"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "bytes",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "grid": {
-
-              },
-              "id": 3,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 1,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 4,
-              "stack": false,
-              "steppedLine": true,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
-                  "hide": false,
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} WAL fsync",
-                  "metric": "etcd_disk_wal_fsync_duration_seconds_bucket",
-                  "refId": "A",
-                  "step": 4
-                },
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} DB fsync",
-                  "metric": "etcd_disk_backend_commit_duration_seconds_bucket",
-                  "refId": "B",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Disk Sync Duration",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "cumulative"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "id": 29,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "process_resident_memory_bytes{job=\"$cluster\"}",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} Resident Memory",
-                  "metric": "process_resident_memory_bytes",
-                  "refId": "A",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "bytes",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "title": "New row"
-        },
-        {
-          "collapse": false,
-          "editable": true,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 5,
-              "id": 22,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 3,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "rate(etcd_network_client_grpc_received_bytes_total{job=\"$cluster\"}[5m])",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} Client Traffic In",
-                  "metric": "etcd_network_client_grpc_received_bytes_total",
-                  "refId": "A",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Client Traffic In",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "Bps",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 5,
-              "id": 21,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 3,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "rate(etcd_network_client_grpc_sent_bytes_total{job=\"$cluster\"}[5m])",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} Client Traffic Out",
-                  "metric": "etcd_network_client_grpc_sent_bytes_total",
-                  "refId": "A",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Client Traffic Out",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "Bps",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "id": 20,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 3,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(etcd_network_peer_received_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} Peer Traffic In",
-                  "metric": "etcd_network_peer_received_bytes_total",
-                  "refId": "A",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Peer Traffic In",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "Bps",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "decimals": null,
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "grid": {
-
-              },
-              "id": 16,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 3,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(etcd_network_peer_sent_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
-                  "hide": false,
-                  "interval": "",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} Peer Traffic Out",
-                  "metric": "etcd_network_peer_sent_bytes_total",
-                  "refId": "A",
-                  "step": 4
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Peer Traffic Out",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "cumulative"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "Bps",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "title": "New row"
-        },
-        {
-          "collapse": false,
-          "editable": true,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "id": 40,
-              "isNew": true,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(etcd_server_proposals_failed_total{job=\"$cluster\"}[5m]))",
-                  "intervalFactor": 2,
-                  "legendFormat": "Proposal Failure Rate",
-                  "metric": "etcd_server_proposals_failed_total",
-                  "refId": "A",
-                  "step": 2
-                },
-                {
-                  "expr": "sum(etcd_server_proposals_pending{job=\"$cluster\"})",
-                  "intervalFactor": 2,
-                  "legendFormat": "Proposal Pending Total",
-                  "metric": "etcd_server_proposals_pending",
-                  "refId": "B",
-                  "step": 2
-                },
-                {
-                  "expr": "sum(rate(etcd_server_proposals_committed_total{job=\"$cluster\"}[5m]))",
-                  "intervalFactor": 2,
-                  "legendFormat": "Proposal Commit Rate",
-                  "metric": "etcd_server_proposals_committed_total",
-                  "refId": "C",
-                  "step": 2
-                },
-                {
-                  "expr": "sum(rate(etcd_server_proposals_applied_total{job=\"$cluster\"}[5m]))",
-                  "intervalFactor": 2,
-                  "legendFormat": "Proposal Apply Rate",
-                  "refId": "D",
-                  "step": 2
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Raft Proposals",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": "",
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "datasource": "$datasource",
-              "decimals": 0,
-              "editable": true,
-              "error": false,
-              "fill": 0,
-              "id": 19,
-              "isNew": true,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": false,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 2,
-              "links": [
-
-              ],
-              "nullPointMode": "connected",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "changes(etcd_server_leader_changes_seen_total{job=\"$cluster\"}[1d])",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} Total Leader Elections Per Day",
-                  "metric": "etcd_server_leader_changes_seen_total",
-                  "refId": "A",
-                  "step": 2
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Total Leader Elections Per Day",
-              "tooltip": {
-                "msResolution": false,
-                "shared": true,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "title": "New row"
-        }
-      ],
-      "schemaVersion": 13,
-      "sharedCrosshair": false,
-      "style": "dark",
-      "tags": [
-
-      ],
-      "templating": {
-        "list": [
-          {
-            "current": {
-              "text": "Prometheus",
-              "value": "Prometheus"
-            },
-            "hide": 0,
-            "label": null,
-            "name": "datasource",
-            "options": [
-
-            ],
-            "query": "prometheus",
-            "refresh": 1,
-            "regex": "",
-            "type": "datasource"
-          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "prod",
-              "value": "prod"
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": "cluster",
-            "multi": false,
-            "name": "cluster",
-            "options": [
-
-            ],
-            "query": "label_values(etcd_server_has_leader, job)",
-            "refresh": 1,
-            "regex": "",
-            "sort": 2,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          }
-        ]
-      },
-      "time": {
-        "from": "now-15m",
-        "to": "now"
-      },
-      "timepicker": {
-        "now": true,
-        "refresh_intervals": [
-          "5s",
-          "10s",
-          "30s",
-          "1m",
-          "5m",
-          "15m",
-          "30m",
-          "1h",
-          "2h",
-          "1d"
-        ],
-        "time_options": [
-          "5m",
-          "15m",
-          "1h",
-          "6h",
-          "12h",
-          "24h",
-          "2d",
-          "7d",
-          "30d"
-        ]
-      },
-      "timezone": "browser",
-      "title": "etcd",
-      "version": 215
-    }
  k8s-cluster-rsrc-use.json: |-
    {
      "annotations": {
@ -1286,7 +62,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:cluster_cpu_utilisation:ratio",
+                  "expr": "node:cluster_cpu_utilisation:ratio{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1372,7 +148,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_cpu_saturation_load1: / scalar(sum(min(kube_pod_info) by (node)))",
+                  "expr": "node:node_cpu_saturation_load1:{cluster=\"$cluster\"} / scalar(sum(min(kube_pod_info{cluster=\"$cluster\"}) by (node)))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1470,7 +246,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:cluster_memory_utilisation:ratio",
+                  "expr": "node:cluster_memory_utilisation:ratio{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1556,7 +332,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_memory_swap_io_bytes:sum_rate",
+                  "expr": "node:node_memory_swap_io_bytes:sum_rate{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1654,7 +430,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_disk_utilisation:avg_irate / scalar(:kube_pod_info_node_count:)",
+                  "expr": "node:node_disk_utilisation:avg_irate{cluster=\"$cluster\"} / scalar(:kube_pod_info_node_count:{cluster=\"$cluster\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1740,7 +516,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_disk_saturation:avg_irate / scalar(:kube_pod_info_node_count:)",
+                  "expr": "node:node_disk_saturation:avg_irate{cluster=\"$cluster\"} / scalar(:kube_pod_info_node_count:{cluster=\"$cluster\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1838,7 +614,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_net_utilisation:sum_irate",
+                  "expr": "node:node_net_utilisation:sum_irate{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -1924,7 +700,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_net_saturation:sum_irate",
+                  "expr": "node:node_net_saturation:sum_irate{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -2022,7 +798,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(max(node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"} - node_filesystem_avail_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"}) by (device,pod,namespace)) by (pod,namespace)\n/ scalar(sum(max(node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\"}) by (device,pod,namespace)))\n* on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:\n",
+                  "expr": "sum(max(node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\", cluster=\"$cluster\"} - node_filesystem_avail_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\", cluster=\"$cluster\"}) by (device,pod,namespace)) by (pod,namespace)\n/ scalar(sum(max(node_filesystem_size_bytes{fstype=~\"ext[234]|btrfs|xfs|zfs\", cluster=\"$cluster\"}) by (device,pod,namespace)))\n* on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{cluster=\"$cluster\"}\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{node}}",
@ -2101,6 +877,33 @@ data:
            "refresh": 1,
            "regex": "",
            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_node_info, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
          }
        ]
      },
@ -2196,7 +999,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_cpu_utilisation:avg1m{node=\"$node\"}",
+                  "expr": "node:node_cpu_utilisation:avg1m{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Utilisation",
@ -2282,7 +1085,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_cpu_saturation_load1:{node=\"$node\"}",
+                  "expr": "node:node_cpu_saturation_load1:{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Saturation",
@ -2380,7 +1183,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_memory_utilisation:{node=\"$node\"}",
+                  "expr": "node:node_memory_utilisation:{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Memory",
@ -2466,7 +1269,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_memory_swap_io_bytes:sum_rate{node=\"$node\"}",
+                  "expr": "node:node_memory_swap_io_bytes:sum_rate{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Swap IO",
@ -2564,7 +1367,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_disk_utilisation:avg_irate{node=\"$node\"}",
+                  "expr": "node:node_disk_utilisation:avg_irate{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Utilisation",
@ -2650,7 +1453,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_disk_saturation:avg_irate{node=\"$node\"}",
+                  "expr": "node:node_disk_saturation:avg_irate{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Saturation",
@ -2748,7 +1551,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_net_utilisation:sum_irate{node=\"$node\"}",
+                  "expr": "node:node_net_utilisation:sum_irate{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Utilisation",
@ -2834,7 +1637,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_net_saturation:sum_irate{node=\"$node\"}",
+                  "expr": "node:node_net_saturation:sum_irate{cluster=\"$cluster\", node=\"$node\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Saturation",
@ -2932,7 +1735,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_filesystem_usage:\n* on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{node=\"$node\"}\n",
+                  "expr": "node:node_filesystem_usage:{cluster=\"$cluster\"}\n* on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{cluster=\"$cluster\", node=\"$node\"}\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}}",
@ -3012,6 +1815,33 @@ data:
            "regex": "",
            "type": "datasource"
          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "prod",
+              "value": "prod"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_node_info, cluster)",
+            "refresh": 1,
+            "regex": "",
+            "sort": 2,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
          {
            "allValue": null,
            "current": {
@ -3027,7 +1857,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_node_info, node)",
+            "query": "label_values(kube_node_info{cluster=\"$cluster\"}, node)",
            "refresh": 1,
            "regex": "",
            "sort": 2,
@ -3075,3199 +1905,6 @@ data:
      "uid": "4ac4f123aae0ff6dbaf4f4f66120033b",
      "version": 0
    }
-  k8s-resources-cluster.json: |-
-    {
-      "annotations": {
-        "list": [
-
-        ]
-      },
-      "editable": true,
-      "gnetId": null,
-      "graphTooltip": 0,
-      "hideControls": false,
-      "links": [
-
-      ],
-      "refresh": "10s",
-      "rows": [
-        {
-          "collapse": false,
-          "height": "100px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "format": "percentunit",
-              "id": 1,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 2,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[1m]))",
-                  "format": "time_series",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "70,80",
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Utilisation",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "singlestat",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "format": "percentunit",
-              "id": 2,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 2,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores) / sum(node:node_num_cpu:sum)",
-                  "format": "time_series",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "70,80",
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Requests Commitment",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "singlestat",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "format": "percentunit",
-              "id": 3,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 2,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores) / sum(node:node_num_cpu:sum)",
-                  "format": "time_series",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "70,80",
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Limits Commitment",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "singlestat",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "format": "percentunit",
-              "id": 4,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 2,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "1 - sum(:node_memory_MemFreeCachedBuffers_bytes:sum) / sum(:node_memory_MemTotal_bytes:sum)",
-                  "format": "time_series",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "70,80",
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Utilisation",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "singlestat",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "format": "percentunit",
-              "id": 5,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 2,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes) / sum(:node_memory_MemTotal_bytes:sum)",
-                  "format": "time_series",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "70,80",
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Requests Commitment",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "singlestat",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "format": "percentunit",
-              "id": 6,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 2,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes) / sum(:node_memory_MemTotal_bytes:sum)",
-                  "format": "time_series",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "70,80",
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Limits Commitment",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "singlestat",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Headlines",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 10,
-              "id": 7,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 0,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate) by (namespace)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{namespace}}",
-                  "legendLink": null,
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Usage",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "CPU",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "id": 8,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "styles": [
-                {
-                  "alias": "Time",
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "pattern": "Time",
-                  "type": "hidden"
-                },
-                {
-                  "alias": "CPU Usage",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #A",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Requests",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #B",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Requests %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #C",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "CPU Limits",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #D",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Limits %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #E",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Namespace",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": true,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-namespace=$__cell",
-                  "pattern": "namespace",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "pattern": "/.*/",
-                  "thresholds": [
-
-                  ],
-                  "type": "string",
-                  "unit": "short"
-                }
-              ],
-              "targets": [
-                {
-                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "B",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate) by (namespace) / sum(kube_pod_container_resource_requests_cpu_cores) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "C",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "D",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate) by (namespace) / sum(kube_pod_container_resource_limits_cpu_cores) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "E",
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Quota",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "transform": "table",
-              "type": "table",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "CPU Quota",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 10,
-              "id": 9,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 0,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(container_memory_rss{container_name!=\"\"}) by (namespace)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{namespace}}",
-                  "legendLink": null,
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Usage (w/o cache)",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "bytes",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Memory",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "id": 10,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "styles": [
-                {
-                  "alias": "Time",
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "pattern": "Time",
-                  "type": "hidden"
-                },
-                {
-                  "alias": "Memory Usage",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #A",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Requests",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #B",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Requests %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #C",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Memory Limits",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #D",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Limits %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #E",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Namespace",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": true,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-namespace=$__cell",
-                  "pattern": "namespace",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "pattern": "/.*/",
-                  "thresholds": [
-
-                  ],
-                  "type": "string",
-                  "unit": "short"
-                }
-              ],
-              "targets": [
-                {
-                  "expr": "sum(container_memory_rss{container_name!=\"\"}) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "B",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(container_memory_rss{container_name!=\"\"}) by (namespace) / sum(kube_pod_container_resource_requests_memory_bytes) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "C",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "D",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(container_memory_rss{container_name!=\"\"}) by (namespace) / sum(kube_pod_container_resource_limits_memory_bytes) by (namespace)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "E",
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Requests by Namespace",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "transform": "table",
-              "type": "table",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Memory Requests",
-          "titleSize": "h6"
-        }
-      ],
-      "schemaVersion": 14,
-      "style": "dark",
-      "tags": [
-        "kubernetes-mixin"
-      ],
-      "templating": {
-        "list": [
-          {
-            "current": {
-              "text": "Prometheus",
-              "value": "Prometheus"
-            },
-            "hide": 0,
-            "label": null,
-            "name": "datasource",
-            "options": [
-
-            ],
-            "query": "prometheus",
-            "refresh": 1,
-            "regex": "",
-            "type": "datasource"
-          }
-        ]
-      },
-      "time": {
-        "from": "now-1h",
-        "to": "now"
-      },
-      "timepicker": {
-        "refresh_intervals": [
-          "5s",
-          "10s",
-          "30s",
-          "1m",
-          "5m",
-          "15m",
-          "30m",
-          "1h",
-          "2h",
-          "1d"
-        ],
-        "time_options": [
-          "5m",
-          "15m",
-          "1h",
-          "6h",
-          "12h",
-          "24h",
-          "2d",
-          "7d",
-          "30d"
-        ]
-      },
-      "timezone": "",
-      "title": "Kubernetes / Compute Resources / Cluster",
-      "uid": "efa86fd1d0c121a26444b636a3f509a8",
-      "version": 0
-    }
-  k8s-resources-namespace.json: |-
-    {
-      "annotations": {
-        "list": [
-
-        ]
-      },
-      "editable": true,
-      "gnetId": null,
-      "graphTooltip": 0,
-      "hideControls": false,
-      "links": [
-
-      ],
-      "refresh": "10s",
-      "rows": [
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 10,
-              "id": 1,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 0,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\"}) by (pod_name)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{pod_name}}",
-                  "legendLink": null,
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Usage",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "CPU Usage",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "id": 2,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "styles": [
-                {
-                  "alias": "Time",
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "pattern": "Time",
-                  "type": "hidden"
-                },
-                {
-                  "alias": "CPU Usage",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #A",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Requests",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #B",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Requests %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #C",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "CPU Limits",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #D",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Limits %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #E",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Pod",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": true,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-namespace=$namespace&var-pod=$__cell",
-                  "pattern": "pod",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "pattern": "/.*/",
-                  "thresholds": [
-
-                  ],
-                  "type": "string",
-                  "unit": "short"
-                }
-              ],
-              "targets": [
-                {
-                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "B",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_requests_cpu_cores{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "C",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "D",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_limits_cpu_cores{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "E",
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Quota",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "transform": "table",
-              "type": "table",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "CPU Quota",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 10,
-              "id": 3,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 0,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(container_memory_usage_bytes{namespace=\"$namespace\", container_name!=\"\"}) by (pod_name)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{pod_name}}",
-                  "legendLink": null,
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Usage (w/o cache)",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "bytes",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Memory Usage",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "id": 4,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "styles": [
-                {
-                  "alias": "Time",
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "pattern": "Time",
-                  "type": "hidden"
-                },
-                {
-                  "alias": "Memory Usage",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #A",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Requests",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #B",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Requests %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #C",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Memory Limits",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #D",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Limits %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #E",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Memory Usage (RSS)",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #F",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Usage (Cache)",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #G",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Usage (Swap",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #H",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Pod",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": true,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-namespace=$namespace&var-pod=$__cell",
-                  "pattern": "pod",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "pattern": "/.*/",
-                  "thresholds": [
-
-                  ],
-                  "type": "string",
-                  "unit": "short"
-                }
-              ],
-              "targets": [
-                {
-                  "expr": "sum(label_replace(container_memory_usage_bytes{namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "B",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_usage_bytes{namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "C",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "D",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_usage_bytes{namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"}) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "E",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_rss{namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "F",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_cache{namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "G",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_swap{namespace=\"$namespace\",container_name!=\"\"}, \"pod\", \"$1\", \"pod_name\", \"(.*)\")) by (pod)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "H",
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Quota",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "transform": "table",
-              "type": "table",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Memory Quota",
-          "titleSize": "h6"
-        }
-      ],
-      "schemaVersion": 14,
-      "style": "dark",
-      "tags": [
-        "kubernetes-mixin"
-      ],
-      "templating": {
-        "list": [
-          {
-            "current": {
-              "text": "Prometheus",
-              "value": "Prometheus"
-            },
-            "hide": 0,
-            "label": null,
-            "name": "datasource",
-            "options": [
-
-            ],
-            "query": "prometheus",
-            "refresh": 1,
-            "regex": "",
-            "type": "datasource"
-          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "prod",
-              "value": "prod"
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": "namespace",
-            "multi": false,
-            "name": "namespace",
-            "options": [
-
-            ],
-            "query": "label_values(kube_pod_info, namespace)",
-            "refresh": 1,
-            "regex": "",
-            "sort": 2,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          }
-        ]
-      },
-      "time": {
-        "from": "now-1h",
-        "to": "now"
-      },
-      "timepicker": {
-        "refresh_intervals": [
-          "5s",
-          "10s",
-          "30s",
-          "1m",
-          "5m",
-          "15m",
-          "30m",
-          "1h",
-          "2h",
-          "1d"
-        ],
-        "time_options": [
-          "5m",
-          "15m",
-          "1h",
-          "6h",
-          "12h",
-          "24h",
-          "2d",
-          "7d",
-          "30d"
-        ]
-      },
-      "timezone": "",
-      "title": "Kubernetes / Compute Resources / Namespace",
-      "uid": "85a562078cdf77779eaa1add43ccec1e",
-      "version": 0
-    }
-  k8s-resources-pod.json: |-
-    {
-      "annotations": {
-        "list": [
-
-        ]
-      },
-      "editable": true,
-      "gnetId": null,
-      "graphTooltip": 0,
-      "hideControls": false,
-      "links": [
-
-      ],
-      "refresh": "10s",
-      "rows": [
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 10,
-              "id": 1,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 0,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\"}) by (container_name)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{container_name}}",
-                  "legendLink": null,
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Usage",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "CPU Usage",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "id": 2,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "styles": [
-                {
-                  "alias": "Time",
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "pattern": "Time",
-                  "type": "hidden"
-                },
-                {
-                  "alias": "CPU Usage",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #A",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Requests",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #B",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Requests %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #C",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "CPU Limits",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #D",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "CPU Limits %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #E",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Container",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "container",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "pattern": "/.*/",
-                  "thresholds": [
-
-                  ],
-                  "type": "string",
-                  "unit": "short"
-                }
-              ],
-              "targets": [
-                {
-                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "B",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod_name=\"$pod\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_requests_cpu_cores{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "C",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "D",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod_name=\"$pod\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_limits_cpu_cores{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "E",
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU Quota",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "transform": "table",
-              "type": "table",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "CPU Quota",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 10,
-              "id": 3,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 0,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": true,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(container_memory_rss{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}) by (container_name)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{container_name}} (RSS)",
-                  "legendLink": null,
-                  "step": 10
-                },
-                {
-                  "expr": "sum(container_memory_cache{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}) by (container_name)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{container_name}} (Cache)",
-                  "legendLink": null,
-                  "step": 10
-                },
-                {
-                  "expr": "sum(container_memory_swap{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}) by (container_name)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{container_name}} (Swap)",
-                  "legendLink": null,
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Usage",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "bytes",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Memory Usage",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "id": 4,
-              "legend": {
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null as zero",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "styles": [
-                {
-                  "alias": "Time",
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "pattern": "Time",
-                  "type": "hidden"
-                },
-                {
-                  "alias": "Memory Usage",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #A",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Requests",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #B",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Requests %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #C",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Memory Limits",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #D",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Limits %",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #E",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "percentunit"
-                },
-                {
-                  "alias": "Memory Usage (RSS)",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #F",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Usage (Cache)",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #G",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Memory Usage (Swap",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "Value #H",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "bytes"
-                },
-                {
-                  "alias": "Container",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "link": false,
-                  "linkTooltip": "Drill down",
-                  "linkUrl": "",
-                  "pattern": "container",
-                  "thresholds": [
-
-                  ],
-                  "type": "number",
-                  "unit": "short"
-                },
-                {
-                  "alias": "",
-                  "colorMode": null,
-                  "colors": [
-
-                  ],
-                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
-                  "decimals": 2,
-                  "pattern": "/.*/",
-                  "thresholds": [
-
-                  ],
-                  "type": "string",
-                  "unit": "short"
-                }
-              ],
-              "targets": [
-                {
-                  "expr": "sum(label_replace(container_memory_usage_bytes{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"POD\", container_name!=\"\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "B",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_usage_bytes{namespace=\"$namespace\", pod_name=\"$pod\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "C",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\", pod=\"$pod\", container!=\"\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "D",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_usage_bytes{namespace=\"$namespace\", pod_name=\"$pod\", container_name!=\"\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "E",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_rss{namespace=\"$namespace\", pod_name=\"$pod\", container_name != \"\", container_name != \"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "F",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_cache{namespace=\"$namespace\", pod_name=\"$pod\", container_name != \"\", container_name != \"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "G",
-                  "step": 10
-                },
-                {
-                  "expr": "sum(label_replace(container_memory_swap{namespace=\"$namespace\", pod_name=\"$pod\", container_name != \"\", container_name != \"POD\"}, \"container\", \"$1\", \"container_name\", \"(.*)\")) by (container)",
-                  "format": "table",
-                  "instant": true,
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "H",
-                  "step": 10
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory Quota",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "transform": "table",
-              "type": "table",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": false
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Memory Quota",
-          "titleSize": "h6"
-        }
-      ],
-      "schemaVersion": 14,
-      "style": "dark",
-      "tags": [
-        "kubernetes-mixin"
-      ],
-      "templating": {
-        "list": [
-          {
-            "current": {
-              "text": "Prometheus",
-              "value": "Prometheus"
-            },
-            "hide": 0,
-            "label": null,
-            "name": "datasource",
-            "options": [
-
-            ],
-            "query": "prometheus",
-            "refresh": 1,
-            "regex": "",
-            "type": "datasource"
-          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "prod",
-              "value": "prod"
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": "namespace",
-            "multi": false,
-            "name": "namespace",
-            "options": [
-
-            ],
-            "query": "label_values(kube_pod_info, namespace)",
-            "refresh": 1,
-            "regex": "",
-            "sort": 2,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "prod",
-              "value": "prod"
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": "pod",
-            "multi": false,
-            "name": "pod",
-            "options": [
-
-            ],
-            "query": "label_values(kube_pod_info{namespace=\"$namespace\"}, pod)",
-            "refresh": 1,
-            "regex": "",
-            "sort": 2,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          }
-        ]
-      },
-      "time": {
-        "from": "now-1h",
-        "to": "now"
-      },
-      "timepicker": {
-        "refresh_intervals": [
-          "5s",
-          "10s",
-          "30s",
-          "1m",
-          "5m",
-          "15m",
-          "30m",
-          "1h",
-          "2h",
-          "1d"
-        ],
-        "time_options": [
-          "5m",
-          "15m",
-          "1h",
-          "6h",
-          "12h",
-          "24h",
-          "2d",
-          "7d",
-          "30d"
-        ]
-      },
-      "timezone": "",
-      "title": "Kubernetes / Compute Resources / Pod",
-      "uid": "6581e46e4e5c7ba40a07646395ef7b23",
-      "version": 0
-    }
  nodes.json: |-
    {
      "__inputs": [
@ -6339,25 +1976,32 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(node_load1{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_load1{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "load 1m",
                  "refId": "A"
                },
                {
-                  "expr": "max(node_load5{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_load5{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "load 5m",
                  "refId": "B"
                },
                {
-                  "expr": "max(node_load15{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_load15{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "load 15m",
                  "refId": "C"
+                },
+                {
+                  "expr": "count(node_cpu_seconds_total{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\", mode=\"user\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "logical cores",
+                  "refId": "D"
                }
              ],
              "thresholds": [
@ -6444,7 +2088,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum by (cpu) (irate(node_cpu_seconds_total{job=\"node-exporter\", mode!=\"idle\", instance=\"$instance\"}[5m]))",
+                  "expr": "sum by (cpu) (irate(node_cpu_seconds_total{cluster=\"$cluster\", job=\"node-exporter\", mode!=\"idle\", instance=\"$instance\"}[5m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{cpu}}",
@ -6548,7 +2192,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max (sum by (cpu) (irate(node_cpu_seconds_total{job=\"node-exporter\", mode!=\"idle\", instance=\"$instance\"}[2m])) ) * 100\n",
+                  "expr": "max (sum by (cpu) (irate(node_cpu_seconds_total{cluster=\"$cluster\", job=\"node-exporter\", mode!=\"idle\", instance=\"$instance\"}[2m])) ) * 100\n",
                  "format": "time_series",
                  "intervalFactor": 10,
                  "legendFormat": "{{ cpu }}",
@ -6560,7 +2204,7 @@ data:
              ],
              "timeFrom": null,
              "timeShift": null,
-              "title": "CPU Utilizaion",
+              "title": "CPU Utilization",
              "tooltip": {
                "shared": false,
                "sort": 0,
@ -6656,7 +2300,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "avg(sum by (cpu) (irate(node_cpu_seconds_total{job=\"node-exporter\", mode!=\"idle\", instance=\"$instance\"}[2m]))) * 100\n",
+                  "expr": "avg(sum by (cpu) (irate(node_cpu_seconds_total{cluster=\"$cluster\", job=\"node-exporter\", mode!=\"idle\", instance=\"$instance\"}[2m]))) * 100\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -6736,28 +2380,28 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(\n  node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"}\n  - node_memory_MemFree_bytes{job=\"node-exporter\", instance=\"$instance\"}\n  - node_memory_Buffers_bytes{job=\"node-exporter\", instance=\"$instance\"}\n  - node_memory_Cached_bytes{job=\"node-exporter\", instance=\"$instance\"}\n)\n",
+                  "expr": "max(\n  node_memory_MemTotal_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n  - node_memory_MemFree_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n  - node_memory_Buffers_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n  - node_memory_Cached_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "memory used",
                  "refId": "A"
                },
                {
-                  "expr": "max(node_memory_Buffers_bytes{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_memory_Buffers_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "memory buffers",
                  "refId": "B"
                },
                {
-                  "expr": "max(node_memory_Cached_bytes{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_memory_Cached_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "memory cached",
                  "refId": "C"
                },
                {
-                  "expr": "max(node_memory_MemFree_bytes{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_memory_MemFree_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "memory free",
@ -6865,7 +2509,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "max(\n  (\n    (\n      node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"}\n    - node_memory_MemFree_bytes{job=\"node-exporter\", instance=\"$instance\"}\n    - node_memory_Buffers_bytes{job=\"node-exporter\", instance=\"$instance\"}\n    - node_memory_Cached_bytes{job=\"node-exporter\", instance=\"$instance\"}\n    )\n    / node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"}\n  ) * 100)\n",
+                  "expr": "max(\n  (\n    (\n      node_memory_MemTotal_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n    - node_memory_MemFree_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n    - node_memory_Buffers_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n    - node_memory_Cached_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n    )\n    / node_memory_MemTotal_bytes{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n  ) * 100)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -6952,21 +2596,21 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(rate(node_disk_read_bytes_total{job=\"node-exporter\", instance=\"$instance\"}[2m]))",
+                  "expr": "max(rate(node_disk_read_bytes_total{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}[2m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "read",
                  "refId": "A"
                },
                {
-                  "expr": "max(rate(node_disk_written_bytes_total{job=\"node-exporter\", instance=\"$instance\"}[2m]))",
+                  "expr": "max(rate(node_disk_written_bytes_total{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}[2m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "written",
                  "refId": "B"
                },
                {
-                  "expr": "max(rate(node_disk_io_time_seconds_total{job=\"node-exporter\",  instance=\"$instance\"}[2m]))",
+                  "expr": "max(rate(node_disk_io_time_seconds_total{cluster=\"$cluster\", job=\"node-exporter\",  instance=\"$instance\"}[2m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "io time",
@ -7057,7 +2701,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "node:node_filesystem_usage:\n",
+                  "expr": "node:node_filesystem_usage:{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}}",
@ -7161,7 +2805,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(rate(node_network_receive_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"lo\"}[5m]))",
+                  "expr": "max(rate(node_network_receive_bytes_total{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\", device!~\"lo\"}[5m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}}",
@ -7252,7 +2896,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(rate(node_network_transmit_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"lo\"}[5m]))",
+                  "expr": "max(rate(node_network_transmit_bytes_total{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\", device!~\"lo\"}[5m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}}",
@ -7356,14 +3000,14 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(\n  node_filesystem_files{job=\"node-exporter\", instance=\"$instance\"}\n  - node_filesystem_files_free{job=\"node-exporter\", instance=\"$instance\"}\n)\n",
+                  "expr": "max(\n  node_filesystem_files{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n  - node_filesystem_files_free{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "inodes used",
                  "refId": "A"
                },
                {
-                  "expr": "max(node_filesystem_files_free{job=\"node-exporter\", instance=\"$instance\"})",
+                  "expr": "max(node_filesystem_files_free{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "inodes free",
@ -7471,7 +3115,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "max(\n  (\n    (\n      node_filesystem_files{job=\"node-exporter\", instance=\"$instance\"}\n    - node_filesystem_files_free{job=\"node-exporter\", instance=\"$instance\"}\n    )\n    / node_filesystem_files{job=\"node-exporter\", instance=\"$instance\"}\n  ) * 100)\n",
+                  "expr": "max(\n  (\n    (\n      node_filesystem_files{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n    - node_filesystem_files_free{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n    )\n    / node_filesystem_files{cluster=\"$cluster\", job=\"node-exporter\", instance=\"$instance\"}\n  ) * 100)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -7531,6 +3175,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
@ -7541,7 +3211,7 @@ data:
            "options": [

            ],
-            "query": "label_values(node_boot_time_seconds{job=\"node-exporter\"}, instance)",
+            "query": "label_values(node_boot_time_seconds{cluster=\"$cluster\", job=\"node-exporter\"}, instance)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -7630,7 +3300,7 @@ data:
              },
              "id": 2,
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": true,
                "current": true,
                "max": true,
@ -7655,16 +3325,23 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
+              "span": 9,
+              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(kubelet_volume_stats_capacity_bytes{job=\"kubelet\", persistentvolumeclaim=\"$volume\"} - kubelet_volume_stats_available_bytes{job=\"kubelet\", persistentvolumeclaim=\"$volume\"}) / kubelet_volume_stats_capacity_bytes{job=\"kubelet\", persistentvolumeclaim=\"$volume\"} * 100\n",
+                  "expr": "(\n  sum without(instance, node) (kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  sum without(instance, node) (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n",
                  "format": "time_series",
                  "intervalFactor": 1,
-                  "legendFormat": "{{ Usage }}",
+                  "legendFormat": "Used Space",
                  "refId": "A"
+                },
+                {
+                  "expr": "sum without(instance, node) (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "Free Space",
+                  "refId": "B"
                }
              ],
              "thresholds": [
@ -7690,22 +3367,106 @@ data:
              },
              "yaxes": [
                {
-                  "format": "percent",
+                  "format": "bytes",
                  "label": null,
                  "logBase": 1,
-                  "max": 100,
+                  "max": null,
                  "min": 0,
                  "show": true
                },
                {
-                  "format": "percent",
+                  "format": "bytes",
                  "label": null,
                  "logBase": 1,
-                  "max": 100,
+                  "max": null,
                  "min": 0,
                  "show": true
                }
              ]
+            },
+            {
+              "cacheTimeout": null,
+              "colorBackground": false,
+              "colorValue": false,
+              "colors": [
+                "rgba(50, 172, 45, 0.97)",
+                "rgba(237, 129, 40, 0.89)",
+                "rgba(245, 54, 54, 0.9)"
+              ],
+              "datasource": "$datasource",
+              "format": "percent",
+              "gauge": {
+                "maxValue": 100,
+                "minValue": 0,
+                "show": true,
+                "thresholdLabels": false,
+                "thresholdMarkers": true
+              },
+              "gridPos": {
+
+              },
+              "id": 3,
+              "interval": null,
+              "links": [
+
+              ],
+              "mappingType": 1,
+              "mappingTypes": [
+                {
+                  "name": "value to text",
+                  "value": 1
+                },
+                {
+                  "name": "range to text",
+                  "value": 2
+                }
+              ],
+              "maxDataPoints": 100,
+              "nullPointMode": "connected",
+              "nullText": null,
+              "postfix": "",
+              "postfixFontSize": "50%",
+              "prefix": "",
+              "prefixFontSize": "50%",
+              "rangeMaps": [
+                {
+                  "from": "null",
+                  "text": "N/A",
+                  "to": "null"
+                }
+              ],
+              "span": 3,
+              "sparkline": {
+                "fillColor": "rgba(31, 118, 189, 0.18)",
+                "full": false,
+                "lineColor": "rgb(31, 120, 193)",
+                "show": false
+              },
+              "tableColumn": "",
+              "targets": [
+                {
+                  "expr": "(\n  kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n  -\n  kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n)\n/\nkubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n* 100\n",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "80, 90",
+              "title": "Volume Space Usage",
+              "tooltip": {
+                "shared": false
+              },
+              "type": "singlestat",
+              "valueFontSize": "80%",
+              "valueMaps": [
+                {
+                  "op": "=",
+                  "text": "N/A",
+                  "value": "null"
+                }
+              ],
+              "valueName": "current"
            }
          ],
          "repeat": null,
@ -7732,9 +3493,9 @@ data:
              "gridPos": {

              },
-              "id": 3,
+              "id": 4,
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": true,
                "current": true,
                "max": true,
@ -7759,16 +3520,23 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
+              "span": 9,
+              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "kubelet_volume_stats_inodes_used{job=\"kubelet\", persistentvolumeclaim=\"$volume\"} / kubelet_volume_stats_inodes{job=\"kubelet\", persistentvolumeclaim=\"$volume\"} * 100\n",
+                  "expr": "sum without(instance, node) (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n",
                  "format": "time_series",
                  "intervalFactor": 1,
-                  "legendFormat": "{{ Usage }}",
+                  "legendFormat": "Used inodes",
                  "refId": "A"
+                },
+                {
+                  "expr": "(\n  sum without(instance, node) (kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  sum without(instance, node) (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": " Free inodes",
+                  "refId": "B"
                }
              ],
              "thresholds": [
@ -7794,22 +3562,106 @@ data:
              },
              "yaxes": [
                {
-                  "format": "percent",
+                  "format": "none",
                  "label": null,
                  "logBase": 1,
-                  "max": 100,
+                  "max": null,
                  "min": 0,
                  "show": true
                },
                {
-                  "format": "percent",
+                  "format": "none",
                  "label": null,
                  "logBase": 1,
-                  "max": 100,
+                  "max": null,
                  "min": 0,
                  "show": true
                }
              ]
+            },
+            {
+              "cacheTimeout": null,
+              "colorBackground": false,
+              "colorValue": false,
+              "colors": [
+                "rgba(50, 172, 45, 0.97)",
+                "rgba(237, 129, 40, 0.89)",
+                "rgba(245, 54, 54, 0.9)"
+              ],
+              "datasource": "$datasource",
+              "format": "percent",
+              "gauge": {
+                "maxValue": 100,
+                "minValue": 0,
+                "show": true,
+                "thresholdLabels": false,
+                "thresholdMarkers": true
+              },
+              "gridPos": {
+
+              },
+              "id": 5,
+              "interval": null,
+              "links": [
+
+              ],
+              "mappingType": 1,
+              "mappingTypes": [
+                {
+                  "name": "value to text",
+                  "value": 1
+                },
+                {
+                  "name": "range to text",
+                  "value": 2
+                }
+              ],
+              "maxDataPoints": 100,
+              "nullPointMode": "connected",
+              "nullText": null,
+              "postfix": "",
+              "postfixFontSize": "50%",
+              "prefix": "",
+              "prefixFontSize": "50%",
+              "rangeMaps": [
+                {
+                  "from": "null",
+                  "text": "N/A",
+                  "to": "null"
+                }
+              ],
+              "span": 3,
+              "sparkline": {
+                "fillColor": "rgba(31, 118, 189, 0.18)",
+                "full": false,
+                "lineColor": "rgb(31, 120, 193)",
+                "show": false
+              },
+              "tableColumn": "",
+              "targets": [
+                {
+                  "expr": "kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n/\nkubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n* 100\n",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "80, 90",
+              "title": "Volume inodes Usage",
+              "tooltip": {
+                "shared": false
+              },
+              "type": "singlestat",
+              "valueFontSize": "80%",
+              "valueMaps": [
+                {
+                  "op": "=",
+                  "text": "N/A",
+                  "value": "null"
+                }
+              ],
+              "valueName": "current"
            }
          ],
          "repeat": null,
@ -7848,6 +3700,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kubelet_volume_stats_capacity_bytes, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
@ -7858,7 +3736,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}, exported_namespace)",
+            "query": "label_values(kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\"}, namespace)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -7884,7 +3762,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kubelet_volume_stats_capacity_bytes{job=\"kubelet\", exported_namespace=\"$namespace\"}, persistentvolumeclaim)",
+            "query": "label_values(kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\"}, persistentvolumeclaim)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -7942,7 +3820,20 @@ data:
      ],
      "annotations": {
        "list": [
-
+          {
+            "builtIn": 1,
+            "datasource": "$datasource",
+            "enable": true,
+            "expr": "time() == BOOL timestamp(rate(kube_pod_container_status_restarts_total{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[2m]) > 0)",
+            "hide": false,
+            "iconColor": "rgba(215, 44, 44, 1)",
+            "name": "Restarts",
+            "showIn": 0,
+            "tags": [
+              "restart"
+            ],
+            "type": "rows"
+          }
        ]
      },
      "editable": false,
@ -7998,29 +3889,37 @@ data:

              ],
              "spaceLength": 10,
+              "span": 12,
              "stack": false,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum by(container_name) (container_memory_usage_bytes{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod_name=\"$pod\", container_name=~\"$container\", container_name!=\"POD\"})",
+                  "expr": "sum by(container_name) (container_memory_usage_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\", container_name=~\"$container\", container_name!=\"POD\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Current: {{ container_name }}",
                  "refId": "A"
                },
                {
-                  "expr": "sum by(container) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\", namespace=\"$namespace\", pod=\"$pod\", container=~\"$container\"})",
+                  "expr": "sum by(container) (kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\", pod=\"$pod\", container=~\"$container\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Requested: {{ container }}",
                  "refId": "B"
                },
                {
-                  "expr": "sum by(container) (kube_pod_container_resource_limits_memory_bytes{job=\"kube-state-metrics\", namespace=\"$namespace\", pod=\"$pod\", container=~\"$container\"})",
+                  "expr": "sum by(container) (kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\", pod=\"$pod\", container=~\"$container\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "Limit: {{ container }}",
                  "refId": "C"
+                },
+                {
+                  "expr": "sum by(container_name) (container_memory_cache{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod_name=~\"$pod\", container_name=~\"$container\", container_name!=\"POD\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Cache: {{ container_name }}",
+                  "refId": "D"
                }
              ],
              "thresholds": [
@ -8115,15 +4014,30 @@ data:

              ],
              "spaceLength": 10,
+              "span": 12,
              "stack": false,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum by (container_name) (rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", image!=\"\",container_name!=\"POD\",pod_name=\"$pod\"}[1m]))",
+                  "expr": "sum by (container_name) (rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", image!=\"\", pod_name=\"$pod\", container_name=~\"$container\", container_name!=\"POD\"}[1m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{ container_name }}",
+                  "legendFormat": "Current: {{ container_name }}",
                  "refId": "A"
+                },
+                {
+                  "expr": "sum by(container) (kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\", pod=\"$pod\", container=~\"$container\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Requested: {{ container }}",
+                  "refId": "B"
+                },
+                {
+                  "expr": "sum by(container) (kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\", pod=\"$pod\", container=~\"$container\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Limit: {{ container }}",
+                  "refId": "C"
                }
              ],
              "thresholds": [
@ -8218,15 +4132,23 @@ data:

              ],
              "spaceLength": 10,
+              "span": 12,
              "stack": false,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sort_desc(sum by (pod_name) (rate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod_name=\"$pod\"}[1m])))",
+                  "expr": "sort_desc(sum by (pod_name) (rate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\"}[1m])))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{ pod_name }}",
+                  "legendFormat": "RX: {{ pod_name }}",
                  "refId": "A"
+                },
+                {
+                  "expr": "sort_desc(sum by (pod_name) (rate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod_name=\"$pod\"}[1m])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "TX: {{ pod_name }}",
+                  "refId": "B"
                }
              ],
              "thresholds": [
@ -8277,6 +4199,110 @@ data:
          "title": "Dashboard Row",
          "titleSize": "h6",
          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "gridPos": {
+
+              },
+              "id": 5,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "max by (container) (kube_pod_container_status_restarts_total{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container=~\"$container\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Restarts: {{ container }}",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Total Restarts Per Container",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": false,
+          "title": "Dashboard Row",
+          "titleSize": "h6",
+          "type": "row"
        }
      ],
      "schemaVersion": 14,
@ -8306,6 +4332,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
@ -8316,7 +4368,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, namespace)",
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -8342,7 +4394,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info{namespace=~\"$namespace\"}, pod)",
+            "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=~\"$namespace\"}, pod)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -8368,7 +4420,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_container_info{namespace=\"$namespace\", pod=\"$pod\"}, container)",
+            "query": "label_values(kube_pod_container_info{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}, container)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -8503,7 +4555,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod_name=~\"$statefulset.*\"}[3m]))",
+                  "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod_name=~\"$statefulset.*\"}[3m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -8586,7 +4638,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod_name=~\"$statefulset.*\"}) / 1024^3",
+                  "expr": "sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod_name=~\"$statefulset.*\"}) / 1024^3",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -8669,7 +4721,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(rate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod_name=~\"$statefulset.*\"}[3m])) + sum(rate(container_network_receive_bytes_total{namespace=\"$namespace\",pod_name=~\"$statefulset.*\"}[3m]))",
+                  "expr": "sum(rate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod_name=~\"$statefulset.*\"}[3m])) + sum(rate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\",pod_name=~\"$statefulset.*\"}[3m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -8767,7 +4819,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
+                  "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -8851,7 +4903,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
+                  "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -8935,7 +4987,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "max(kube_statefulset_status_observed_generation{job=\"kube-state-metrics\",  namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
+                  "expr": "max(kube_statefulset_status_observed_generation{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -9019,7 +5071,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "max(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", statefulset=\"$statefulset\", namespace=\"$namespace\"}) without (instance, pod)",
+                  "expr": "max(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -9098,35 +5150,35 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\",namespace=\"$namespace\"}) without (instance, pod)",
+                  "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "replicas specified",
                  "refId": "A"
                },
                {
-                  "expr": "max(kube_statefulset_status_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\",namespace=\"$namespace\"}) without (instance, pod)",
+                  "expr": "max(kube_statefulset_status_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "replicas created",
                  "refId": "B"
                },
                {
-                  "expr": "min(kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\", statefulset=\"$statefulset\",namespace=\"$namespace\"}) without (instance, pod)",
+                  "expr": "min(kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "ready",
                  "refId": "C"
                },
                {
-                  "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", statefulset=\"$statefulset\",namespace=\"$namespace\"}) without (instance, pod)",
+                  "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "replicas of current version",
                  "refId": "D"
                },
                {
-                  "expr": "min(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\", statefulset=\"$statefulset\",namespace=\"$namespace\"}) without (instance, pod)",
+                  "expr": "min(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "updated",
@ -9210,6 +5262,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_statefulset_metadata_generation, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -23,20 +23,30 @@ spec:
    spec:
      containers:
        - name: grafana
-          image: grafana/grafana:6.0.0
+          image: grafana/grafana:6.2.4
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
          ports:
            - name: http
              containerPort: 8080
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: 8080
+            initialDelaySeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /api/health
+              port: 8080
+            initialDelaySeconds: 10
          resources:
            requests:
-              memory: 100Mi
              cpu: 100m
+              memory: 100Mi
            limits:
-              memory: 200Mi
              cpu: 200m
+              memory: 200Mi
          volumeMounts:
            - name: config
              mountPath: /etc/grafana
@ -44,8 +54,12 @@ spec:
              mountPath: /etc/grafana/provisioning/datasources
            - name: providers
              mountPath: /etc/grafana/provisioning/dashboards
-            - name: dashboards
-              mountPath: /etc/grafana/dashboards
+            - name: dashboards-etcd
+              mountPath: /etc/grafana/dashboards/etcd
+            - name: dashboards-k8s
+              mountPath: /etc/grafana/dashboards/k8s
+            - name: dashboards-k8s-resources
+              mountPath: /etc/grafana/dashboards/k8s-resources
      volumes:
        - name: config
          configMap:
@ -56,6 +70,13 @@ spec:
        - name: providers
          configMap:
            name: grafana-providers
-        - name: dashboards
+        - name: dashboards-etcd
          configMap:
-            name: grafana-dashboards
+            name: grafana-dashboards-etcd
+        - name: dashboards-k8s
+          configMap:
+            name: grafana-dashboards-k8s
+        - name: dashboards-k8s-resources
+          configMap:
+            name: grafana-dashboards-k8s-resources
+
--- a/addons/heapster/cluster-role-binding.yaml
+++ b/addons/heapster/cluster-role-binding.yaml
@ -1,12 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: heapster
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: heapster
-subjects:
- kind: ServiceAccount
-  name: heapster
-  namespace: kube-system
--- a/addons/heapster/cluster-role.yaml
+++ b/addons/heapster/cluster-role.yaml
@ -1,30 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: heapster
-rules:
- apiGroups:
-  - ""
-  resources:
-  - events
-  - namespaces
-  - nodes
-  - pods
-  verbs:
-  - get
-  - list
-  - watch
- apiGroups:
-  - extensions
-  resources:
-  - deployments
-  verbs:
-  - get
-  - list
-  - watch
- apiGroups:
-  - ""
-  resources:
-  - nodes/stats
-  verbs:
-  - get
--- a/addons/heapster/deployment.yaml
+++ b/addons/heapster/deployment.yaml
@ -1,62 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: heapster
-  namespace: kube-system
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      name: heapster
-      phase: prod
-  template:
-    metadata:
-      labels:
-        name: heapster
-        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
-    spec:
-      serviceAccountName: heapster
-      containers:
-        - name: heapster
-          image: k8s.gcr.io/heapster-amd64:v1.5.4
-          command:
-            - /heapster
-            - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8082
-              scheme: HTTP
-            initialDelaySeconds: 180
-            timeoutSeconds: 5
-        - name: heapster-nanny
-          image: k8s.gcr.io/addon-resizer:1.7
-          command:
-            - /pod_nanny
-            - --cpu=80m
-            - --extra-cpu=0.5m
-            - --memory=140Mi
-            - --extra-memory=4Mi
-            - --threshold=5
-            - --deployment=heapster
-            - --container=heapster
-            - --poll-period=300000
-            - --estimator=exponential
-          env:
-            - name: MY_POD_NAME
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.name
-            - name: MY_POD_NAMESPACE
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.namespace
-          resources:
-            limits:
-              cpu: 50m
-              memory: 90Mi
-            requests:
-              cpu: 50m
-              memory: 90Mi
--- a/addons/heapster/role-binding.yaml
+++ b/addons/heapster/role-binding.yaml
@ -1,13 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: heapster
-  namespace: kube-system
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: system:pod-nanny
-subjects:
- kind: ServiceAccount
-  name: heapster
-  namespace: kube-system
--- a/addons/heapster/role.yaml
+++ b/addons/heapster/role.yaml
@ -1,19 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: system:pod-nanny
-  namespace: kube-system
-rules:
- apiGroups:
-  - ""
-  resources:
-  - pods
-  verbs:
-  - get
- apiGroups:
-  - "extensions"
-  resources:
-  - deployments
-  verbs:
-  - get
-  - update
--- a/addons/heapster/service-account.yaml
+++ b/addons/heapster/service-account.yaml
@ -1,5 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: heapster
-  namespace: kube-system
--- a/addons/heapster/service.yaml
+++ b/addons/heapster/service.yaml
@ -1,12 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata: 
-  name: heapster
-  namespace: kube-system
-spec: 
-  type: ClusterIP
-  selector:
-    name: heapster
-  ports: 
-    - port: 80
-      targetPort: 8082
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/aws/rbac/role.yaml
+++ b/addons/nginx-ingress/aws/rbac/role.yaml
@ -37,5 +37,3 @@ rules:
      - endpoints
    verbs:
      - get
-      - create
-      - update
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/azure/rbac/role.yaml
+++ b/addons/nginx-ingress/azure/rbac/role.yaml
@ -37,5 +37,3 @@ rules:
      - endpoints
    verbs:
      - get
-      - create
-      - update
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -22,7 +22,7 @@ spec:
    spec:
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/bare-metal/rbac/role.yaml
+++ b/addons/nginx-ingress/bare-metal/rbac/role.yaml
@ -37,5 +37,3 @@ rules:
      - endpoints
    verbs:
      - get
-      - create
-      - update
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/digital-ocean/rbac/role.yaml
+++ b/addons/nginx-ingress/digital-ocean/rbac/role.yaml
@ -37,5 +37,3 @@ rules:
      - endpoints
    verbs:
      - get
-      - create
-      - update
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -24,7 +24,7 @@ spec:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
+          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --ingress-class=public
--- a/addons/nginx-ingress/google-cloud/rbac/role.yaml
+++ b/addons/nginx-ingress/google-cloud/rbac/role.yaml
@ -37,5 +37,3 @@ rules:
      - endpoints
    verbs:
      - get
-      - create
-      - update
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -20,7 +20,7 @@ spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.7.1
+          image: quay.io/prometheus/prometheus:v2.10.0
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
@ -28,6 +28,10 @@ spec:
          ports:
            - name: web
              containerPort: 9090
+          resources:
+            requests:
+              cpu: 100m
+              memory: 200Mi
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
@ -35,18 +39,18 @@ spec:
              mountPath: /etc/prometheus/rules
            - name: data
              mountPath: /var/lib/prometheus
-          readinessProbe:
-            httpGet:
-              path: /-/ready
-              port: 9090
-            initialDelaySeconds: 10
-            timeoutSeconds: 10
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
            initialDelaySeconds: 10
            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /-/ready
+              port: 9090
+            initialDelaySeconds: 10
+            timeoutSeconds: 10
      terminationGracePeriodSeconds: 30
      volumes:
        - name: config
--- a/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
@ -27,6 +27,7 @@ rules:
  - daemonsets
  - deployments
  - replicasets
+  - ingresses
  verbs:
  - list
  - watch
@ -62,3 +63,10 @@ rules:
  verbs:
  - list
  - watch
+- apiGroups:
+  - certificates.k8s.io
+  resources:
+  - certificatesigningrequests
+  verbs:
+  - list
+  - watch
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -24,7 +24,7 @@ spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.5.0
+        image: quay.io/coreos/kube-state-metrics:v1.6.0
        ports:
          - name: metrics
            containerPort: 8080
@ -35,7 +35,7 @@ spec:
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer
-        image: k8s.gcr.io/addon-resizer:1.8.4
+        image: k8s.gcr.io/addon-resizer:1.8.5
        resources:
          limits:
            cpu: 100m
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -28,7 +28,7 @@ spec:
      hostPID: true
      containers:
      - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v0.17.0
+        image: quay.io/prometheus/node-exporter:v0.18.1
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
--- a/addons/prometheus/network-policy.yaml
+++ b/addons/prometheus/network-policy.yaml
@ -0,0 +1,28 @@
+# Allow Grafana access and in-cluster Prometheus scraping
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: prometheus
+  namespace: monitoring
+spec:
+  podSelector:
+    matchLabels:
+      name: prometheus
+  ingress:
+    - ports:
+        - protocol: TCP
+          port: 9090
+      from:
+        - namespaceSelector:
+            matchLabels:
+              name: monitoring
+          podSelector:
+            matchLabels:
+              name: grafana
+        - namespaceSelector:
+            matchLabels:
+              name: monitoring
+          podSelector:
+            matchLabels:
+              name: prometheus
+
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -183,12 +183,33 @@ data:
              "record": "namespace_name:container_memory_usage_bytes:sum"
            },
            {
-              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"}) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"} * on (endpoint, instance, job, namespace, pod, service) group_left(phase) (kube_pod_status_phase{phase=~\"^(Pending|Running)$\"} == 1)) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
              "record": "namespace_name:kube_pod_container_resource_requests_memory_bytes:sum"
            },
            {
-              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"} and on(pod) kube_pod_status_scheduled{condition=\"true\"}) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
+              "expr": "sum by (namespace, label_name) (\n  sum(kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"} * on (endpoint, instance, job, namespace, pod, service) group_left(phase) (kube_pod_status_phase{phase=~\"^(Pending|Running)$\"} == 1)) by (namespace, pod)\n* on (namespace, pod) group_left(label_name)\n  label_replace(kube_pod_labels{job=\"kube-state-metrics\"}, \"pod_name\", \"$1\", \"pod\", \"(.*)\")\n)\n",
              "record": "namespace_name:kube_pod_container_resource_requests_cpu_cores:sum"
+            },
+            {
+              "expr": "sum(\n  label_replace(\n    label_replace(\n      kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"ReplicaSet\"},\n      \"replicaset\", \"$1\", \"owner_name\", \"(.*)\"\n    ) * on(replicaset, namespace) group_left(owner_name) kube_replicaset_owner{job=\"kube-state-metrics\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n) by (namespace, workload, pod)\n",
+              "labels": {
+                "workload_type": "deployment"
+              },
+              "record": "mixin_pod_workload"
+            },
+            {
+              "expr": "sum(\n  label_replace(\n    kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"DaemonSet\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n) by (namespace, workload, pod)\n",
+              "labels": {
+                "workload_type": "daemonset"
+              },
+              "record": "mixin_pod_workload"
+            },
+            {
+              "expr": "sum(\n  label_replace(\n    kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"StatefulSet\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n) by (namespace, workload, pod)\n",
+              "labels": {
+                "workload_type": "statefulset"
+              },
+              "record": "mixin_pod_workload"
            }
          ]
        },
@ -366,19 +387,19 @@ data:
              "record": "node:node_memory_swap_io_bytes:sum_rate"
            },
            {
-              "expr": "avg(irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]))\n",
+              "expr": "avg(irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+\"}[1m]))\n",
              "record": ":node_disk_utilisation:avg_irate"
            },
            {
-              "expr": "avg by (node) (\n  irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "expr": "avg by (node) (\n  irate(node_disk_io_time_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
              "record": "node:node_disk_utilisation:avg_irate"
            },
            {
-              "expr": "avg(irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]) / 1e3)\n",
+              "expr": "avg(irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+\"}[1m]))\n",
              "record": ":node_disk_saturation:avg_irate"
            },
            {
-              "expr": "avg by (node) (\n  irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+\"}[1m]) / 1e3\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
+              "expr": "avg by (node) (\n  irate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\",device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+\"}[1m])\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n)\n",
              "record": "node:node_disk_saturation:avg_irate"
            },
            {
@ -662,7 +683,7 @@ data:
                "message": "Cluster has overcommitted CPU resource requests for Namespaces.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
              },
-              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"requests.cpu\"})\n  /\nsum(node:node_num_cpu:sum)\n  > 1.5\n",
+              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"cpu\"})\n  /\nsum(node:node_num_cpu:sum)\n  > 1.5\n",
              "for": "5m",
              "labels": {
                "severity": "warning"
@ -674,7 +695,7 @@ data:
                "message": "Cluster has overcommitted memory resource requests for Namespaces.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
              },
-              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"requests.memory\"})\n  /\nsum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n  > 1.5\n",
+              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"memory\"})\n  /\nsum(node_memory_MemTotal_bytes{job=\"node-exporter\"})\n  > 1.5\n",
              "for": "5m",
              "labels": {
                "severity": "warning"
@ -840,7 +861,7 @@ data:
                "message": "API server is returning errors for {{ $value }}% of requests.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
              },
-              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) without(instance, pod)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) without(instance, pod) * 100 > 10\n",
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m]))\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) * 100 > 3\n",
              "for": "10m",
              "labels": {
                "severity": "critical"
@ -852,7 +873,31 @@ data:
                "message": "API server is returning errors for {{ $value }}% of requests.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
              },
-              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) without(instance, pod)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) without(instance, pod) * 100 > 5\n",
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m]))\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) * 100 > 1\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorsHigh",
+              "annotations": {
+                "message": "API server is returning errors for {{ $value }}% of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
+              },
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) by (resource,subresource,verb)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) by (resource,subresource,verb) * 100 > 10\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorsHigh",
+              "annotations": {
+                "message": "API server is returning errors for {{ $value }}% of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
+              },
+              "expr": "sum(rate(apiserver_request_count{job=\"apiserver\",code=~\"^(?:5..)$\"}[5m])) by (resource,subresource,verb)\n  /\nsum(rate(apiserver_request_count{job=\"apiserver\"}[5m])) by (resource,subresource,verb) * 100 > 5\n",
              "for": "10m",
              "labels": {
                "severity": "warning"
@ -861,10 +906,10 @@ data:
            {
              "alert": "KubeClientCertificateExpiration",
              "annotations": {
-                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 7 days.",
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 7.0 days.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
              },
-              "expr": "histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 604800\n",
+              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 604800\n",
              "labels": {
                "severity": "warning"
              }
@ -872,10 +917,10 @@ data:
            {
              "alert": "KubeClientCertificateExpiration",
              "annotations": {
-                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 24 hours.",
+                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours.",
                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
              },
-              "expr": "histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 86400\n",
+              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 86400\n",
              "labels": {
                "severity": "critical"
              }
@ -947,6 +992,60 @@ data:
            }
          ]
        },
+        {
+          "name": "node-time",
+          "rules": [
+            {
+              "alert": "ClockSkewDetected",
+              "annotations": {
+                "message": "Clock skew detected on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}. Ensure NTP is configured correctly on this host."
+              },
+              "expr": "abs(node_timex_offset_seconds{job=\"node-exporter\"}) > 0.03\n",
+              "for": "2m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "name": "node-network",
+          "rules": [
+            {
+              "alert": "NetworkReceiveErrors",
+              "annotations": {
+                "message": "Network interface \"{{ $labels.device }}\" showing receive errors on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\""
+              },
+              "expr": "rate(node_network_receive_errs_total{job=\"node-exporter\",device!~\"veth.+|tunl.+\"}[2m]) > 0\n",
+              "for": "2m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NetworkTransmitErrors",
+              "annotations": {
+                "message": "Network interface \"{{ $labels.device }}\" showing transmit errors on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\""
+              },
+              "expr": "rate(node_network_transmit_errs_total{job=\"node-exporter\",device!~\"veth.+|tunl.+\"}[2m]) > 0\n",
+              "for": "2m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeNetworkInterfaceFlapping",
+              "annotations": {
+                "message": "Network interface \"{{ $labels.device }}\" changing it's up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\""
+              },
+              "expr": "changes(node_network_up{job=\"node-exporter\",device!~\"veth.+|tunl.+\"}[2m]) > 2\n",
+              "for": "2m",
+              "labels": {
+                "severity": "warning"
+              }
+            }
+          ]
+        },
        {
          "name": "prometheus.rules",
          "rules": [
@ -1040,7 +1139,7 @@ data:
                "description": "{{$labels.job}} at {{$labels.instance}} has a corrupted write-ahead log (WAL).",
                "summary": "Prometheus write-ahead log is corrupted"
              },
-              "expr": "tsdb_wal_corruptions_total{job=\"prometheus\"} > 0\n",
+              "expr": "prometheus_tsdb_wal_corruptions_total{job=\"prometheus\"} > 0\n",
              "for": "4h",
              "labels": {
                "severity": "warning"
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/aws/container-linux/kubernetes/ami.tf
+++ b/aws/container-linux/kubernetes/ami.tf
@ -2,10 +2,10 @@ locals {
  # Pick a CoreOS Container Linux derivative
  # coreos-stable -> Container Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id

-  flavor  = "${element(split("-", var.os_image), 0)}"
-  channel = "${element(split("-", var.os_image), 1)}"
+  flavor  = element(split("-", var.os_image), 0)
+  channel = element(split("-", var.os_image), 1)
 }

 data "aws_ami" "coreos" {
@ -24,7 +24,7 @@ data "aws_ami" "coreos" {

  filter {
    name   = "name"
-    values = ["CoreOS-${local.channel}-*"]
+    values = ["CoreOS-${local.flavor == "coreos" ? local.channel : "stable"}-*"]
  }
 }

@ -44,6 +44,7 @@ data "aws_ami" "flatcar" {

  filter {
    name   = "name"
-    values = ["Flatcar-${local.channel}-*"]
+    values = ["Flatcar-${local.flavor == "flatcar" ? local.channel : "stable"}-*"]
  }
 }
+
--- a/aws/container-linux/kubernetes/bootkube.tf
+++ b/aws/container-linux/kubernetes/bootkube.tf
@ -1,15 +1,17 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"

-  cluster_name          = "${var.cluster_name}"
-  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers          = ["${aws_route53_record.etcds.*.fqdn}"]
-  asset_dir             = "${var.asset_dir}"
-  networking            = "${var.networking}"
-  network_mtu           = "${var.network_mtu}"
-  pod_cidr              = "${var.pod_cidr}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  enable_reporting      = "${var.enable_reporting}"
+  cluster_name          = var.cluster_name
+  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
+  etcd_servers          = aws_route53_record.etcds.*.fqdn
+  asset_dir             = var.asset_dir
+  networking            = var.networking
+  network_mtu           = var.network_mtu
+  pod_cidr              = var.pod_cidr
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  enable_reporting      = var.enable_reporting
+  enable_aggregation    = var.enable_aggregation
 }
+
--- a/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/aws/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.12"
+            Environment="ETCD_IMAGE_TAG=v3.3.13"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -63,6 +63,7 @@ systemd:
          --volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
          --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@ -77,6 +78,7 @@ systemd:
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -123,7 +125,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.4
+          KUBELET_IMAGE_TAG=v1.15.0
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -1,87 +1,90 @@
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "aws_route53_record" "etcds" {
-  count = "${var.controller_count}"
+  count = var.controller_count

  # DNS Zone where record should be created
-  zone_id = "${var.dns_zone_id}"
+  zone_id = var.dns_zone_id

-  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)}"
+  name = format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)
  type = "A"
  ttl  = 300

  # private IPv4 address for etcd
-  records = ["${element(aws_instance.controllers.*.private_ip, count.index)}"]
+  records = [element(aws_instance.controllers.*.private_ip, count.index)]
 }

 # Controller instances
 resource "aws_instance" "controllers" {
-  count = "${var.controller_count}"
+  count = var.controller_count

  tags = {
    Name = "${var.cluster_name}-controller-${count.index}"
  }

-  instance_type = "${var.controller_type}"
+  instance_type = var.controller_type

-  ami       = "${local.ami_id}"
-  user_data = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  ami       = local.ami_id
+  user_data = element(data.ct_config.controller-ignitions.*.rendered, count.index)

  # storage
  root_block_device {
-    volume_type = "${var.disk_type}"
-    volume_size = "${var.disk_size}"
-    iops        = "${var.disk_iops}"
+    volume_type = var.disk_type
+    volume_size = var.disk_size
+    iops        = var.disk_iops
  }

  # network
  associate_public_ip_address = true
-  subnet_id                   = "${element(aws_subnet.public.*.id, count.index)}"
-  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
+  subnet_id                   = element(aws_subnet.public.*.id, count.index)
+  vpc_security_group_ids      = [aws_security_group.controller.id]

  lifecycle {
    ignore_changes = [
-      "ami",
-      "user_data",
+      ami,
+      user_data,
    ]
  }
 }

 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  count = var.controller_count
+  content = element(
+    data.template_file.controller-configs.*.rendered,
+    count.index,
+  )
  pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
+  snippets     = var.controller_clc_snippets
 }

 # Controller Container Linux configs
 data "template_file" "controller-configs" {
-  count = "${var.controller_count}"
+  count = var.controller_count

-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
-
-    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    cgroup_driver          = local.flavor == "flatcar" && local.channel == "edge" ? "systemd" : "cgroupfs"
+    kubeconfig             = indent(10, module.bootkube.kubeconfig-kubelet)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
  }
 }

 data "template_file" "etcds" {
-  count    = "${var.controller_count}"
+  count    = var.controller_count
  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"

-  vars {
-    index        = "${count.index}"
-    cluster_name = "${var.cluster_name}"
-    dns_zone     = "${var.dns_zone}"
+  vars = {
+    index        = count.index
+    cluster_name = var.cluster_name
+    dns_zone     = var.dns_zone
  }
 }
+
--- a/aws/container-linux/kubernetes/network.tf
+++ b/aws/container-linux/kubernetes/network.tf
@ -1,57 +1,67 @@
-data "aws_availability_zones" "all" {}
+data "aws_availability_zones" "all" {
+}

 # Network VPC, gateway, and routes

 resource "aws_vpc" "network" {
-  cidr_block                       = "${var.host_cidr}"
+  cidr_block                       = var.host_cidr
  assign_generated_ipv6_cidr_block = true
  enable_dns_support               = true
  enable_dns_hostnames             = true

-  tags = "${map("Name", "${var.cluster_name}")}"
+  tags = {
+    "Name" = var.cluster_name
+  }
 }

 resource "aws_internet_gateway" "gateway" {
-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id

-  tags = "${map("Name", "${var.cluster_name}")}"
+  tags = {
+    "Name" = var.cluster_name
+  }
 }

 resource "aws_route_table" "default" {
-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id

  route {
    cidr_block = "0.0.0.0/0"
-    gateway_id = "${aws_internet_gateway.gateway.id}"
+    gateway_id = aws_internet_gateway.gateway.id
  }

  route {
    ipv6_cidr_block = "::/0"
-    gateway_id      = "${aws_internet_gateway.gateway.id}"
+    gateway_id      = aws_internet_gateway.gateway.id
  }

-  tags = "${map("Name", "${var.cluster_name}")}"
+  tags = {
+    "Name" = var.cluster_name
+  }
 }

 # Subnets (one per availability zone)

 resource "aws_subnet" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
+  count = length(data.aws_availability_zones.all.names)

-  vpc_id            = "${aws_vpc.network.id}"
-  availability_zone = "${data.aws_availability_zones.all.names[count.index]}"
+  vpc_id            = aws_vpc.network.id
+  availability_zone = data.aws_availability_zones.all.names[count.index]

-  cidr_block                      = "${cidrsubnet(var.host_cidr, 4, count.index)}"
-  ipv6_cidr_block                 = "${cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)}"
+  cidr_block                      = cidrsubnet(var.host_cidr, 4, count.index)
+  ipv6_cidr_block                 = cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)
  map_public_ip_on_launch         = true
  assign_ipv6_address_on_creation = true

-  tags = "${map("Name", "${var.cluster_name}-public-${count.index}")}"
+  tags = {
+    "Name" = "${var.cluster_name}-public-${count.index}"
+  }
 }

 resource "aws_route_table_association" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
+  count = length(data.aws_availability_zones.all.names)

-  route_table_id = "${aws_route_table.default.id}"
-  subnet_id      = "${element(aws_subnet.public.*.id, count.index)}"
+  route_table_id = aws_route_table.default.id
+  subnet_id      = element(aws_subnet.public.*.id, count.index)
 }
+
--- a/aws/container-linux/kubernetes/nlb.tf
+++ b/aws/container-linux/kubernetes/nlb.tf
@ -1,14 +1,14 @@
 # Network Load Balancer DNS Record
 resource "aws_route53_record" "apiserver" {
-  zone_id = "${var.dns_zone_id}"
+  zone_id = var.dns_zone_id

-  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
+  name = format("%s.%s.", var.cluster_name, var.dns_zone)
  type = "A"

  # AWS recommends their special "alias" records for NLBs
  alias {
-    name                   = "${aws_lb.nlb.dns_name}"
-    zone_id                = "${aws_lb.nlb.zone_id}"
+    name                   = aws_lb.nlb.dns_name
+    zone_id                = aws_lb.nlb.zone_id
    evaluate_target_health = true
  }
 }
@ -19,51 +19,51 @@ resource "aws_lb" "nlb" {
  load_balancer_type = "network"
  internal           = false

-  subnets = ["${aws_subnet.public.*.id}"]
+  subnets = aws_subnet.public.*.id

  enable_cross_zone_load_balancing = true
 }

 # Forward TCP apiserver traffic to controllers
 resource "aws_lb_listener" "apiserver-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
+  load_balancer_arn = aws_lb.nlb.arn
  protocol          = "TCP"
  port              = "6443"

  default_action {
    type             = "forward"
-    target_group_arn = "${aws_lb_target_group.controllers.arn}"
+    target_group_arn = aws_lb_target_group.controllers.arn
  }
 }

 # Forward HTTP ingress traffic to workers
 resource "aws_lb_listener" "ingress-http" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
+  load_balancer_arn = aws_lb.nlb.arn
  protocol          = "TCP"
  port              = 80

  default_action {
    type             = "forward"
-    target_group_arn = "${module.workers.target_group_http}"
+    target_group_arn = module.workers.target_group_http
  }
 }

 # Forward HTTPS ingress traffic to workers
 resource "aws_lb_listener" "ingress-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
+  load_balancer_arn = aws_lb.nlb.arn
  protocol          = "TCP"
  port              = 443

  default_action {
    type             = "forward"
-    target_group_arn = "${module.workers.target_group_https}"
+    target_group_arn = module.workers.target_group_https
  }
 }

 # Target group of controllers
 resource "aws_lb_target_group" "controllers" {
  name        = "${var.cluster_name}-controllers"
-  vpc_id      = "${aws_vpc.network.id}"
+  vpc_id      = aws_vpc.network.id
  target_type = "instance"

  protocol = "TCP"
@ -85,9 +85,10 @@ resource "aws_lb_target_group" "controllers" {

 # Attach controller instances to apiserver NLB
 resource "aws_lb_target_group_attachment" "controllers" {
-  count = "${var.controller_count}"
+  count = var.controller_count

-  target_group_arn = "${aws_lb_target_group.controllers.arn}"
-  target_id        = "${element(aws_instance.controllers.*.id, count.index)}"
+  target_group_arn = aws_lb_target_group.controllers.arn
+  target_id        = element(aws_instance.controllers.*.id, count.index)
  port             = 6443
 }
+
--- a/aws/container-linux/kubernetes/outputs.tf
+++ b/aws/container-linux/kubernetes/outputs.tf
@ -1,48 +1,54 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }

 # Outputs for Kubernetes Ingress

 output "ingress_dns_name" {
-  value       = "${aws_lb.nlb.dns_name}"
+  value       = aws_lb.nlb.dns_name
  description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
 }

 output "ingress_zone_id" {
-  value       = "${aws_lb.nlb.zone_id}"
+  value       = aws_lb.nlb.zone_id
  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
 }

 # Outputs for worker pools

 output "vpc_id" {
-  value       = "${aws_vpc.network.id}"
+  value       = aws_vpc.network.id
  description = "ID of the VPC for creating worker instances"
 }

 output "subnet_ids" {
-  value       = ["${aws_subnet.public.*.id}"]
+  value       = aws_subnet.public.*.id
  description = "List of subnet IDs for creating worker instances"
 }

 output "worker_security_groups" {
-  value       = ["${aws_security_group.worker.id}"]
+  value       = [aws_security_group.worker.id]
  description = "List of worker security group IDs"
 }

 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig-kubelet}"
+  value = module.bootkube.kubeconfig-kubelet
 }

 # Outputs for custom load balancing

+output "nlb_id" {
+  description = "ARN of the Network Load Balancer"
+  value       = aws_lb.nlb.id
+}
+
 output "worker_target_group_http" {
  description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${module.workers.target_group_http}"
+  value       = module.workers.target_group_http
 }

 output "worker_target_group_https" {
  description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${module.workers.target_group_https}"
+  value       = module.workers.target_group_https
 }
+
--- a/aws/container-linux/kubernetes/require.tf
+++ b/aws/container-linux/kubernetes/require.tf
@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "aws" {
-  version = "~> 1.13"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}
--- a/aws/container-linux/kubernetes/security.tf
+++ b/aws/container-linux/kubernetes/security.tf
@ -6,13 +6,15 @@ resource "aws_security_group" "controller" {
  name        = "${var.cluster_name}-controller"
  description = "${var.cluster_name} controller security group"

-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id

-  tags = "${map("Name", "${var.cluster_name}-controller")}"
+  tags = {
+    "Name" = "${var.cluster_name}-controller"
+  }
 }

 resource "aws_security_group_rule" "controller-ssh" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type        = "ingress"
  protocol    = "tcp"
@ -22,7 +24,7 @@ resource "aws_security_group_rule" "controller-ssh" {
 }

 resource "aws_security_group_rule" "controller-etcd" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type      = "ingress"
  protocol  = "tcp"
@ -33,17 +35,41 @@ resource "aws_security_group_rule" "controller-etcd" {

 # Allow Prometheus to scrape etcd metrics
 resource "aws_security_group_rule" "controller-etcd-metrics" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 2381
  to_port                  = 2381
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
+}
+
+resource "aws_security_group_rule" "controller-vxlan" {
+  count = var.networking == "flannel" ? 1 : 0
+
+  security_group_id = aws_security_group.controller.id
+
+  type                     = "ingress"
+  protocol                 = "udp"
+  from_port                = 4789
+  to_port                  = 4789
+  source_security_group_id = aws_security_group.worker.id
+}
+
+resource "aws_security_group_rule" "controller-vxlan-self" {
+  count = var.networking == "flannel" ? 1 : 0
+
+  security_group_id = aws_security_group.controller.id
+
+  type      = "ingress"
+  protocol  = "udp"
+  from_port = 4789
+  to_port   = 4789
+  self      = true
 }

 resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type        = "ingress"
  protocol    = "tcp"
@ -52,50 +78,30 @@ resource "aws_security_group_rule" "controller-apiserver" {
  cidr_blocks = ["0.0.0.0/0"]
 }

-resource "aws_security_group_rule" "controller-flannel" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "udp"
-  from_port                = 8472
-  to_port                  = 8472
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-flannel-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "udp"
-  from_port = 8472
-  to_port   = 8472
-  self      = true
-}
-
 # Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "controller-node-exporter" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 9100
  to_port                  = 9100
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }

 # Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "controller-kubelet" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 10250
  to_port                  = 10250
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }

 resource "aws_security_group_rule" "controller-kubelet-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type      = "ingress"
  protocol  = "tcp"
@ -105,17 +111,17 @@ resource "aws_security_group_rule" "controller-kubelet-self" {
 }

 resource "aws_security_group_rule" "controller-bgp" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 179
  to_port                  = 179
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }

 resource "aws_security_group_rule" "controller-bgp-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type      = "ingress"
  protocol  = "tcp"
@ -125,17 +131,17 @@ resource "aws_security_group_rule" "controller-bgp-self" {
 }

 resource "aws_security_group_rule" "controller-ipip" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type                     = "ingress"
  protocol                 = 4
  from_port                = 0
  to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }

 resource "aws_security_group_rule" "controller-ipip-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type      = "ingress"
  protocol  = 4
@ -145,17 +151,17 @@ resource "aws_security_group_rule" "controller-ipip-self" {
 }

 resource "aws_security_group_rule" "controller-ipip-legacy" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type                     = "ingress"
  protocol                 = 94
  from_port                = 0
  to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
+  source_security_group_id = aws_security_group.worker.id
 }

 resource "aws_security_group_rule" "controller-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type      = "ingress"
  protocol  = 94
@ -165,7 +171,7 @@ resource "aws_security_group_rule" "controller-ipip-legacy-self" {
 }

 resource "aws_security_group_rule" "controller-egress" {
-  security_group_id = "${aws_security_group.controller.id}"
+  security_group_id = aws_security_group.controller.id

  type             = "egress"
  protocol         = "-1"
@ -181,13 +187,15 @@ resource "aws_security_group" "worker" {
  name        = "${var.cluster_name}-worker"
  description = "${var.cluster_name} worker security group"

-  vpc_id = "${aws_vpc.network.id}"
+  vpc_id = aws_vpc.network.id

-  tags = "${map("Name", "${var.cluster_name}-worker")}"
+  tags = {
+    "Name" = "${var.cluster_name}-worker"
+  }
 }

 resource "aws_security_group_rule" "worker-ssh" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type        = "ingress"
  protocol    = "tcp"
@ -197,7 +205,7 @@ resource "aws_security_group_rule" "worker-ssh" {
 }

 resource "aws_security_group_rule" "worker-http" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type        = "ingress"
  protocol    = "tcp"
@ -207,7 +215,7 @@ resource "aws_security_group_rule" "worker-http" {
 }

 resource "aws_security_group_rule" "worker-https" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type        = "ingress"
  protocol    = "tcp"
@ -216,29 +224,33 @@ resource "aws_security_group_rule" "worker-https" {
  cidr_blocks = ["0.0.0.0/0"]
 }

-resource "aws_security_group_rule" "worker-flannel" {
-  security_group_id = "${aws_security_group.worker.id}"
+resource "aws_security_group_rule" "worker-vxlan" {
+  count = var.networking == "flannel" ? 1 : 0
+
+  security_group_id = aws_security_group.worker.id

  type                     = "ingress"
  protocol                 = "udp"
-  from_port                = 8472
-  to_port                  = 8472
-  source_security_group_id = "${aws_security_group.controller.id}"
+  from_port                = 4789
+  to_port                  = 4789
+  source_security_group_id = aws_security_group.controller.id
 }

-resource "aws_security_group_rule" "worker-flannel-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+resource "aws_security_group_rule" "worker-vxlan-self" {
+  count = var.networking == "flannel" ? 1 : 0
+
+  security_group_id = aws_security_group.worker.id

  type      = "ingress"
  protocol  = "udp"
-  from_port = 8472
-  to_port   = 8472
+  from_port = 4789
+  to_port   = 4789
  self      = true
 }

 # Allow Prometheus to scrape node-exporter daemonset
 resource "aws_security_group_rule" "worker-node-exporter" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type      = "ingress"
  protocol  = "tcp"
@ -248,7 +260,7 @@ resource "aws_security_group_rule" "worker-node-exporter" {
 }

 resource "aws_security_group_rule" "ingress-health" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type        = "ingress"
  protocol    = "tcp"
@ -259,18 +271,18 @@ resource "aws_security_group_rule" "ingress-health" {

 # Allow apiserver to access kubelets for exec, log, port-forward
 resource "aws_security_group_rule" "worker-kubelet" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 10250
  to_port                  = 10250
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }

 # Allow Prometheus to scrape kubelet metrics
 resource "aws_security_group_rule" "worker-kubelet-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type      = "ingress"
  protocol  = "tcp"
@ -280,17 +292,17 @@ resource "aws_security_group_rule" "worker-kubelet-self" {
 }

 resource "aws_security_group_rule" "worker-bgp" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 179
  to_port                  = 179
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }

 resource "aws_security_group_rule" "worker-bgp-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type      = "ingress"
  protocol  = "tcp"
@ -300,17 +312,17 @@ resource "aws_security_group_rule" "worker-bgp-self" {
 }

 resource "aws_security_group_rule" "worker-ipip" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type                     = "ingress"
  protocol                 = 4
  from_port                = 0
  to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }

 resource "aws_security_group_rule" "worker-ipip-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type      = "ingress"
  protocol  = 4
@ -320,17 +332,17 @@ resource "aws_security_group_rule" "worker-ipip-self" {
 }

 resource "aws_security_group_rule" "worker-ipip-legacy" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type                     = "ingress"
  protocol                 = 94
  from_port                = 0
  to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
+  source_security_group_id = aws_security_group.controller.id
 }

 resource "aws_security_group_rule" "worker-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type      = "ingress"
  protocol  = 94
@ -340,7 +352,7 @@ resource "aws_security_group_rule" "worker-ipip-legacy-self" {
 }

 resource "aws_security_group_rule" "worker-egress" {
-  security_group_id = "${aws_security_group.worker.id}"
+  security_group_id = aws_security_group.worker.id

  type             = "egress"
  protocol         = "-1"
@ -349,3 +361,4 @@ resource "aws_security_group_rule" "worker-egress" {
  cidr_blocks      = ["0.0.0.0/0"]
  ipv6_cidr_blocks = ["::/0"]
 }
+
--- a/aws/container-linux/kubernetes/ssh.tf
+++ b/aws/container-linux/kubernetes/ssh.tf
@ -1,46 +1,46 @@
 # Secure copy etcd TLS assets to controllers.
 resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
+  count = var.controller_count

  connection {
    type    = "ssh"
-    host    = "${element(aws_instance.controllers.*.public_ip, count.index)}"
+    host    = element(aws_instance.controllers.*.public_ip, count.index)
    user    = "core"
    timeout = "15m"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
    destination = "$HOME/etcd-client-ca.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
    destination = "$HOME/etcd-client.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
    destination = "$HOME/etcd-client.key"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
    destination = "$HOME/etcd-server.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
    destination = "$HOME/etcd-server.key"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
    destination = "$HOME/etcd-peer.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
    destination = "$HOME/etcd-peer.key"
  }

@ -64,21 +64,21 @@ resource "null_resource" "copy-controller-secrets" {
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
  depends_on = [
-    "module.bootkube",
-    "module.workers",
-    "aws_route53_record.apiserver",
-    "null_resource.copy-controller-secrets",
+    module.bootkube,
+    module.workers,
+    aws_route53_record.apiserver,
+    null_resource.copy-controller-secrets,
  ]

  connection {
    type    = "ssh"
-    host    = "${aws_instance.controllers.0.public_ip}"
+    host    = aws_instance.controllers[0].public_ip
    user    = "core"
    timeout = "15m"
  }

  provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
    destination = "$HOME/assets"
  }

@ -89,3 +89,4 @@ resource "null_resource" "bootkube-start" {
    ]
  }
 }
+
--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -1,84 +1,90 @@
 variable "cluster_name" {
-  type        = "string"
+  type        = string
  description = "Unique cluster name (prepended to dns_zone)"
 }

 # AWS

 variable "dns_zone" {
-  type        = "string"
+  type        = string
  description = "AWS Route53 DNS Zone (e.g. aws.example.com)"
 }

 variable "dns_zone_id" {
-  type        = "string"
+  type        = string
  description = "AWS Route53 DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
 }

 # instances

 variable "controller_count" {
-  type        = "string"
+  type        = string
  default     = "1"
  description = "Number of controllers (i.e. masters)"
 }

 variable "worker_count" {
-  type        = "string"
+  type        = string
  default     = "1"
  description = "Number of workers"
 }

 variable "controller_type" {
-  type        = "string"
+  type        = string
  default     = "t3.small"
  description = "EC2 instance type for controllers"
 }

 variable "worker_type" {
-  type        = "string"
+  type        = string
  default     = "t3.small"
  description = "EC2 instance type for workers"
 }

 variable "os_image" {
-  type        = "string"
+  type        = string
  default     = "coreos-stable"
-  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
 }

 variable "disk_size" {
-  type        = "string"
+  type        = string
  default     = "40"
  description = "Size of the EBS volume in GB"
 }

 variable "disk_type" {
-  type        = "string"
+  type        = string
  default     = "gp2"
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

 variable "disk_iops" {
-  type        = "string"
+  type        = string
  default     = "0"
  description = "IOPS of the EBS volume (e.g. 100)"
 }

 variable "worker_price" {
-  type        = "string"
+  type        = string
  default     = ""
  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
 }

+variable "worker_target_groups" {
+  type        = list(string)
+  description = "Additional target group ARNs to which worker instances should be added"
+  default     = []
+}
+
 variable "controller_clc_snippets" {
-  type        = "list"
+  type        = list(string)
  description = "Controller Container Linux Config snippets"
  default     = []
 }

 variable "worker_clc_snippets" {
-  type        = "list"
+  type        = list(string)
  description = "Worker Container Linux Config snippets"
  default     = []
 }
@ -86,36 +92,36 @@ variable "worker_clc_snippets" {
 # configuration

 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
  description = "SSH public key for user 'core'"
 }

 variable "asset_dir" {
  description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
+  type        = string
 }

 variable "networking" {
  description = "Choice of networking provider (calico or flannel)"
-  type        = "string"
+  type        = string
  default     = "calico"
 }

 variable "network_mtu" {
  description = "CNI interface MTU (applies to calico only). Use 8981 if using instances types with Jumbo frames."
-  type        = "string"
+  type        = string
  default     = "1480"
 }

 variable "host_cidr" {
  description = "CIDR IPv4 range to assign to EC2 nodes"
-  type        = "string"
+  type        = string
  default     = "10.0.0.0/16"
 }

 variable "pod_cidr" {
  description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
+  type        = string
  default     = "10.2.0.0/16"
 }

@ -125,18 +131,26 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD

-  type    = "string"
+
+  type = string
  default = "10.3.0.0/16"
 }

 variable "cluster_domain_suffix" {
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
+  type = string
+  default = "cluster.local"
 }

 variable "enable_reporting" {
-  type        = "string"
+  type = string
  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
+  default = "false"
 }
+
+variable "enable_aggregation" {
+  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  type = string
+  default = "false"
+}
+
--- a/aws/container-linux/kubernetes/versions.tf
+++ b/aws/container-linux/kubernetes/versions.tf
@ -0,0 +1,11 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.0"
+  required_providers {
+    aws      = "~> 2.7"
+    ct       = "~> 0.3.2"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}
--- a/aws/container-linux/kubernetes/workers.tf
+++ b/aws/container-linux/kubernetes/workers.tf
@ -1,21 +1,23 @@
 module "workers" {
-  source = "workers"
-  name   = "${var.cluster_name}"
+  source = "./workers"
+  name   = var.cluster_name

  # AWS
-  vpc_id          = "${aws_vpc.network.id}"
-  subnet_ids      = ["${aws_subnet.public.*.id}"]
-  security_groups = ["${aws_security_group.worker.id}"]
-  count           = "${var.worker_count}"
-  instance_type   = "${var.worker_type}"
-  os_image        = "${var.os_image}"
-  disk_size       = "${var.disk_size}"
-  spot_price      = "${var.worker_price}"
+  vpc_id          = aws_vpc.network.id
+  subnet_ids      = aws_subnet.public.*.id
+  security_groups = [aws_security_group.worker.id]
+  worker_count    = var.worker_count
+  instance_type   = var.worker_type
+  os_image        = var.os_image
+  disk_size       = var.disk_size
+  spot_price      = var.worker_price
+  target_groups   = var.worker_target_groups

  # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
-  ssh_authorized_key    = "${var.ssh_authorized_key}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  clc_snippets          = "${var.worker_clc_snippets}"
+  kubeconfig            = module.bootkube.kubeconfig-kubelet
+  ssh_authorized_key    = var.ssh_authorized_key
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  clc_snippets          = var.worker_clc_snippets
 }
+
--- a/aws/container-linux/kubernetes/workers/ami.tf
+++ b/aws/container-linux/kubernetes/workers/ami.tf
@ -2,10 +2,10 @@ locals {
  # Pick a CoreOS Container Linux derivative
  # coreos-stable -> Container Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id = "${local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id}"
+  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id

-  flavor  = "${element(split("-", var.os_image), 0)}"
-  channel = "${element(split("-", var.os_image), 1)}"
+  flavor  = element(split("-", var.os_image), 0)
+  channel = element(split("-", var.os_image), 1)
 }

 data "aws_ami" "coreos" {
@ -24,7 +24,7 @@ data "aws_ami" "coreos" {

  filter {
    name   = "name"
-    values = ["CoreOS-${local.channel}-*"]
+    values = ["CoreOS-${local.flavor == "coreos" ? local.channel : "stable"}-*"]
  }
 }

@ -44,6 +44,7 @@ data "aws_ami" "flatcar" {

  filter {
    name   = "name"
-    values = ["Flatcar-${local.channel}-*"]
+    values = ["Flatcar-${local.flavor == "flatcar" ? local.channel : "stable"}-*"]
  }
 }
+
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -38,6 +38,7 @@ systemd:
          --volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
          --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@ -50,6 +51,7 @@ systemd:
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -93,7 +95,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.4
+          KUBELET_IMAGE_TAG=v1.15.0
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -111,7 +113,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.13.4 \
+            docker://k8s.gcr.io/hyperkube:v1.15.0 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
--- a/aws/container-linux/kubernetes/workers/ingress.tf
+++ b/aws/container-linux/kubernetes/workers/ingress.tf
@ -2,7 +2,7 @@

 resource "aws_lb_target_group" "workers-http" {
  name        = "${var.name}-workers-http"
-  vpc_id      = "${var.vpc_id}"
+  vpc_id      = var.vpc_id
  target_type = "instance"

  protocol = "TCP"
@ -25,7 +25,7 @@ resource "aws_lb_target_group" "workers-http" {

 resource "aws_lb_target_group" "workers-https" {
  name        = "${var.name}-workers-https"
-  vpc_id      = "${var.vpc_id}"
+  vpc_id      = var.vpc_id
  target_type = "instance"

  protocol = "TCP"
@ -45,3 +45,4 @@ resource "aws_lb_target_group" "workers-https" {
    interval = 10
  }
 }
+
--- a/aws/container-linux/kubernetes/workers/outputs.tf
+++ b/aws/container-linux/kubernetes/workers/outputs.tf
@ -1,9 +1,10 @@
 output "target_group_http" {
  description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${aws_lb_target_group.workers-http.arn}"
+  value       = aws_lb_target_group.workers-http.arn
 }

 output "target_group_https" {
  description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${aws_lb_target_group.workers-https.arn}"
+  value       = aws_lb_target_group.workers-https.arn
 }
+
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -1,71 +1,77 @@
 variable "name" {
-  type        = "string"
+  type        = string
  description = "Unique name for the worker pool"
 }

 # AWS

 variable "vpc_id" {
-  type        = "string"
+  type        = string
  description = "Must be set to `vpc_id` output by cluster"
 }

 variable "subnet_ids" {
-  type        = "list"
+  type        = list(string)
  description = "Must be set to `subnet_ids` output by cluster"
 }

 variable "security_groups" {
-  type        = "list"
+  type        = list(string)
  description = "Must be set to `worker_security_groups` output by cluster"
 }

 # instances

-variable "count" {
-  type        = "string"
+variable "worker_count" {
+  type        = string
  default     = "1"
  description = "Number of instances"
 }

 variable "instance_type" {
-  type        = "string"
+  type        = string
  default     = "t3.small"
  description = "EC2 instance type"
 }

 variable "os_image" {
-  type        = "string"
+  type        = string
  default     = "coreos-stable"
-  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha)"
+  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
 }

 variable "disk_size" {
-  type        = "string"
+  type        = string
  default     = "40"
  description = "Size of the EBS volume in GB"
 }

 variable "disk_type" {
-  type        = "string"
+  type        = string
  default     = "gp2"
  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
 }

 variable "disk_iops" {
-  type        = "string"
+  type        = string
  default     = "0"
  description = "IOPS of the EBS volume (required for io1)"
 }

 variable "spot_price" {
-  type        = "string"
+  type        = string
  default     = ""
  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
 }

+variable "target_groups" {
+  type        = list(string)
+  description = "Additional target group ARNs to which instances should be added"
+  default     = []
+}
+
 variable "clc_snippets" {
-  type        = "list"
+  type        = list(string)
  description = "Container Linux Config snippets"
  default     = []
 }
@ -73,12 +79,12 @@ variable "clc_snippets" {
 # configuration

 variable "kubeconfig" {
-  type        = "string"
+  type        = string
  description = "Must be set to `kubeconfig` output by cluster"
 }

 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
  description = "SSH public key for user 'core'"
 }

@ -88,12 +94,14 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD

-  type    = "string"
+
+  type = string
  default = "10.3.0.0/16"
 }

 variable "cluster_domain_suffix" {
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
+  type = string
+  default = "cluster.local"
 }
+
--- a/aws/container-linux/kubernetes/workers/versions.tf
+++ b/aws/container-linux/kubernetes/workers/versions.tf
@ -0,0 +1,4 @@
+
+terraform {
+  required_version = ">= 0.12"
+}
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -3,23 +3,24 @@ resource "aws_autoscaling_group" "workers" {
  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"

  # count
-  desired_capacity          = "${var.count}"
-  min_size                  = "${var.count}"
-  max_size                  = "${var.count + 2}"
+  desired_capacity          = var.worker_count
+  min_size                  = var.worker_count
+  max_size                  = var.worker_count + 2
  default_cooldown          = 30
  health_check_grace_period = 30

  # network
-  vpc_zone_identifier = ["${var.subnet_ids}"]
+  vpc_zone_identifier = var.subnet_ids

  # template
-  launch_configuration = "${aws_launch_configuration.worker.name}"
+  launch_configuration = aws_launch_configuration.worker.name

  # target groups to which instances should be added
-  target_group_arns = [
-    "${aws_lb_target_group.workers-http.id}",
-    "${aws_lb_target_group.workers-https.id}",
-  ]
+  target_group_arns = flatten([
+    aws_lb_target_group.workers-http.id,
+    aws_lb_target_group.workers-https.id,
+    var.target_groups,
+  ])

  lifecycle {
    # override the default destroy and replace update behavior
@ -32,54 +33,58 @@ resource "aws_autoscaling_group" "workers" {
  # used. Disable wait to avoid issues and align with other clouds.
  wait_for_capacity_timeout = "0"

-  tags = [{
-    key                 = "Name"
-    value               = "${var.name}-worker"
-    propagate_at_launch = true
-  }]
+  tags = [
+    {
+      key                 = "Name"
+      value               = "${var.name}-worker"
+      propagate_at_launch = true
+    },
+  ]
 }

 # Worker template
 resource "aws_launch_configuration" "worker" {
-  image_id          = "${local.ami_id}"
-  instance_type     = "${var.instance_type}"
-  spot_price        = "${var.spot_price}"
+  image_id          = local.ami_id
+  instance_type     = var.instance_type
+  spot_price        = var.spot_price
  enable_monitoring = false

-  user_data = "${data.ct_config.worker-ignition.rendered}"
+  user_data = data.ct_config.worker-ignition.rendered

  # storage
  root_block_device {
-    volume_type = "${var.disk_type}"
-    volume_size = "${var.disk_size}"
-    iops        = "${var.disk_iops}"
+    volume_type = var.disk_type
+    volume_size = var.disk_size
+    iops        = var.disk_iops
  }

  # network
-  security_groups = ["${var.security_groups}"]
+  security_groups = var.security_groups

  lifecycle {
    // Override the default destroy and replace update behavior
    create_before_destroy = true
-    ignore_changes        = ["image_id"]
+    ignore_changes        = [image_id]
  }
 }

 # Worker Ignition config
 data "ct_config" "worker-ignition" {
-  content      = "${data.template_file.worker-config.rendered}"
+  content      = data.template_file.worker-config.rendered
  pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
+  snippets     = var.clc_snippets
 }

 # Worker Container Linux config
 data "template_file" "worker-config" {
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+  template = file("${path.module}/cl/worker.yaml.tmpl")

  vars = {
-    kubeconfig             = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    kubeconfig             = indent(10, var.kubeconfig)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    cgroup_driver          = local.flavor == "flatcar" && local.channel == "edge" ? "systemd" : "cgroupfs"
  }
 }
+
--- a/aws/fedora-atomic/kubernetes/LICENSE
+++ b/aws/fedora-atomic/kubernetes/LICENSE
@ -1,23 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2017 Typhoon Authors
-Copyright (c) 2017 Dalton Hubble
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
-
--- a/aws/fedora-atomic/kubernetes/README.md
+++ b/aws/fedora-atomic/kubernetes/README.md
@ -1,23 +0,0 @@
-# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
-
-Typhoon is a minimal and free Kubernetes distribution.
-
-* Minimal, stable base Kubernetes distribution
-* Declarative infrastructure and configuration
-* Free (freedom and cost) and privacy-respecting
-* Practical for labs, datacenters, and clouds
-
-Typhoon distributes upstream Kubernetes, architectural conventions, and cluster addons, much like a GNU/Linux distribution provides the Linux kernel and userspace components.
-
-## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>
-
-* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/) and [spot](https://typhoon.psdn.io/cl/aws/#spot) workers
-* Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)
-
-## Docs
-
-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
-
--- a/aws/fedora-atomic/kubernetes/ami.tf
+++ b/aws/fedora-atomic/kubernetes/ami.tf
@ -1,19 +0,0 @@
-data "aws_ami" "fedora" {
-  most_recent = true
-  owners      = ["125523088429"]
-
-  filter {
-    name   = "architecture"
-    values = ["x86_64"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "name"
-    values = ["Fedora-AtomicHost-28-20180625.1.x86_64-*-gp2-*"]
-  }
-}
--- a/aws/fedora-atomic/kubernetes/bootkube.tf
+++ b/aws/fedora-atomic/kubernetes/bootkube.tf
@ -1,18 +0,0 @@
-# Self-hosted Kubernetes assets (kubeconfig, manifests)
-module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
-
-  cluster_name          = "${var.cluster_name}"
-  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers          = ["${aws_route53_record.etcds.*.fqdn}"]
-  asset_dir             = "${var.asset_dir}"
-  networking            = "${var.networking}"
-  network_mtu           = "${var.network_mtu}"
-  pod_cidr              = "${var.pod_cidr}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  enable_reporting      = "${var.enable_reporting}"
-
-  # Fedora
-  trusted_certs_dir = "/etc/pki/tls/certs"
-}
--- a/aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
+++ b/aws/fedora-atomic/kubernetes/cloudinit/controller.yaml.tmpl
@ -1,93 +0,0 @@
-#cloud-config
-write_files:
-  - path: /etc/etcd/etcd.conf
-    content: |
-      ETCD_NAME=${etcd_name}
-      ETCD_DATA_DIR=/var/lib/etcd
-      ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
-      ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
-      ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
-      ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
-      ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
-      ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
-      ETCD_STRICT_RECONFIG_CHECK=true
-      ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
-      ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
-      ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
-      ETCD_CLIENT_CERT_AUTH=true
-      ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
-      ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
-      ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
-      ETCD_PEER_CLIENT_CERT_AUTH=true
-  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
-    content: |
-      [Unit]
-      Wants=rpc-statd.service
-      [Service]
-      ExecStartPre=/bin/mkdir -p /opt/cni/bin
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/inactive-manifests
-      ExecStartPre=/bin/mkdir -p /var/lib/cni
-      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-      Restart=always
-      RestartSec=10
-  - path: /etc/kubernetes/kubelet.conf
-    content: |
-      ARGS="--anonymous-auth=false \
-        --authentication-token-webhook \
-        --authorization-mode=Webhook \
-        --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${cluster_dns_service_ip} \
-        --cluster_domain=${cluster_domain_suffix} \
-        --cni-conf-dir=/etc/kubernetes/cni/net.d \
-        --exit-on-lock-contention \
-        --kubeconfig=/etc/kubernetes/kubeconfig \
-        --lock-file=/var/run/lock/kubelet.lock \
-        --network-plugin=cni \
-        --node-labels=node-role.kubernetes.io/master \
-        --node-labels=node-role.kubernetes.io/controller="true" \
-        --pod-manifest-path=/etc/kubernetes/manifests \
-        --read-only-port=0 \
-        --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
-        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
-  - path: /etc/kubernetes/kubeconfig
-    permissions: '0644'
-    content: |
-      ${kubeconfig}
-  - path: /var/lib/bootkube/.keep
-  - path: /etc/NetworkManager/conf.d/typhoon.conf
-    content: |
-      [main]
-      plugins=keyfile
-      [keyfile]
-      unmanaged-devices=interface-name:cali*;interface-name:tunl*
-  - path: /etc/selinux/config
-    owner: root:root
-    permissions: '0644'
-    content: |
-      SELINUX=permissive
-      SELINUXTYPE=targeted
-bootcmd:
-  - [setenforce, Permissive]
-  - [systemctl, disable, firewalld, --now]
-  # https://github.com/kubernetes/kubernetes/issues/60869
-  - [modprobe, ip_vs]
-runcmd:
-  - [systemctl, daemon-reload]
-  - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=etcd quay.io/poseidon/etcd:v3.3.12"
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
-  - "atomic install --system --name=bootkube quay.io/poseidon/bootkube:v0.14.0"
-  - [systemctl, start, --no-block, etcd.service]
-  - [systemctl, start, --no-block, kubelet.service]
-users:
-  - default
-  - name: fedora
-    gecos: Fedora Admin
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: wheel,adm,systemd-journal,docker
-    ssh-authorized-keys:
-      - "${ssh_authorized_key}"
--- a/aws/fedora-atomic/kubernetes/controllers.tf
+++ b/aws/fedora-atomic/kubernetes/controllers.tf
@ -1,79 +0,0 @@
-# Discrete DNS records for each controller's private IPv4 for etcd usage
-resource "aws_route53_record" "etcds" {
-  count = "${var.controller_count}"
-
-  # DNS Zone where record should be created
-  zone_id = "${var.dns_zone_id}"
-
-  name = "${format("%s-etcd%d.%s.", var.cluster_name, count.index, var.dns_zone)}"
-  type = "A"
-  ttl  = 300
-
-  # private IPv4 address for etcd
-  records = ["${element(aws_instance.controllers.*.private_ip, count.index)}"]
-}
-
-# Controller instances
-resource "aws_instance" "controllers" {
-  count = "${var.controller_count}"
-
-  tags = {
-    Name = "${var.cluster_name}-controller-${count.index}"
-  }
-
-  instance_type = "${var.controller_type}"
-
-  ami       = "${data.aws_ami.fedora.image_id}"
-  user_data = "${element(data.template_file.controller-cloudinit.*.rendered, count.index)}"
-
-  # storage
-  root_block_device {
-    volume_type = "${var.disk_type}"
-    volume_size = "${var.disk_size}"
-    iops        = "${var.disk_iops}"
-  }
-
-  # network
-  associate_public_ip_address = true
-  subnet_id                   = "${element(aws_subnet.public.*.id, count.index)}"
-  vpc_security_group_ids      = ["${aws_security_group.controller.id}"]
-
-  lifecycle {
-    ignore_changes = [
-      "ami",
-      "user_data",
-    ]
-  }
-}
-
-# Controller Cloud-Init
-data "template_file" "controller-cloudinit" {
-  count = "${var.controller_count}"
-
-  template = "${file("${path.module}/cloudinit/controller.yaml.tmpl")}"
-
-  vars = {
-    # Cannot use cyclic dependencies on controllers or their DNS records
-    etcd_name   = "etcd${count.index}"
-    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
-    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
-
-    kubeconfig             = "${indent(6, module.bootkube.kubeconfig-kubelet)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-  }
-}
-
-data "template_file" "etcds" {
-  count    = "${var.controller_count}"
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars {
-    index        = "${count.index}"
-    cluster_name = "${var.cluster_name}"
-    dns_zone     = "${var.dns_zone}"
-  }
-}
--- a/aws/fedora-atomic/kubernetes/network.tf
+++ b/aws/fedora-atomic/kubernetes/network.tf
@ -1,57 +0,0 @@
-data "aws_availability_zones" "all" {}
-
-# Network VPC, gateway, and routes
-
-resource "aws_vpc" "network" {
-  cidr_block                       = "${var.host_cidr}"
-  assign_generated_ipv6_cidr_block = true
-  enable_dns_support               = true
-  enable_dns_hostnames             = true
-
-  tags = "${map("Name", "${var.cluster_name}")}"
-}
-
-resource "aws_internet_gateway" "gateway" {
-  vpc_id = "${aws_vpc.network.id}"
-
-  tags = "${map("Name", "${var.cluster_name}")}"
-}
-
-resource "aws_route_table" "default" {
-  vpc_id = "${aws_vpc.network.id}"
-
-  route {
-    cidr_block = "0.0.0.0/0"
-    gateway_id = "${aws_internet_gateway.gateway.id}"
-  }
-
-  route {
-    ipv6_cidr_block = "::/0"
-    gateway_id      = "${aws_internet_gateway.gateway.id}"
-  }
-
-  tags = "${map("Name", "${var.cluster_name}")}"
-}
-
-# Subnets (one per availability zone)
-
-resource "aws_subnet" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
-
-  vpc_id            = "${aws_vpc.network.id}"
-  availability_zone = "${data.aws_availability_zones.all.names[count.index]}"
-
-  cidr_block                      = "${cidrsubnet(var.host_cidr, 4, count.index)}"
-  ipv6_cidr_block                 = "${cidrsubnet(aws_vpc.network.ipv6_cidr_block, 8, count.index)}"
-  map_public_ip_on_launch         = true
-  assign_ipv6_address_on_creation = true
-
-  tags = "${map("Name", "${var.cluster_name}-public-${count.index}")}"
-}
-
-resource "aws_route_table_association" "public" {
-  count = "${length(data.aws_availability_zones.all.names)}"
-
-  route_table_id = "${aws_route_table.default.id}"
-  subnet_id      = "${element(aws_subnet.public.*.id, count.index)}"
-}
--- a/aws/fedora-atomic/kubernetes/nlb.tf
+++ b/aws/fedora-atomic/kubernetes/nlb.tf
@ -1,93 +0,0 @@
-# Network Load Balancer DNS Record
-resource "aws_route53_record" "apiserver" {
-  zone_id = "${var.dns_zone_id}"
-
-  name = "${format("%s.%s.", var.cluster_name, var.dns_zone)}"
-  type = "A"
-
-  # AWS recommends their special "alias" records for NLBs
-  alias {
-    name                   = "${aws_lb.nlb.dns_name}"
-    zone_id                = "${aws_lb.nlb.zone_id}"
-    evaluate_target_health = true
-  }
-}
-
-# Network Load Balancer for apiservers and ingress
-resource "aws_lb" "nlb" {
-  name               = "${var.cluster_name}-nlb"
-  load_balancer_type = "network"
-  internal           = false
-
-  subnets = ["${aws_subnet.public.*.id}"]
-
-  enable_cross_zone_load_balancing = true
-}
-
-# Forward TCP apiserver traffic to controllers
-resource "aws_lb_listener" "apiserver-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
-  protocol          = "TCP"
-  port              = "6443"
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${aws_lb_target_group.controllers.arn}"
-  }
-}
-
-# Forward HTTP ingress traffic to workers
-resource "aws_lb_listener" "ingress-http" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
-  protocol          = "TCP"
-  port              = 80
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${module.workers.target_group_http}"
-  }
-}
-
-# Forward HTTPS ingress traffic to workers
-resource "aws_lb_listener" "ingress-https" {
-  load_balancer_arn = "${aws_lb.nlb.arn}"
-  protocol          = "TCP"
-  port              = 443
-
-  default_action {
-    type             = "forward"
-    target_group_arn = "${module.workers.target_group_https}"
-  }
-}
-
-# Target group of controllers
-resource "aws_lb_target_group" "controllers" {
-  name        = "${var.cluster_name}-controllers"
-  vpc_id      = "${aws_vpc.network.id}"
-  target_type = "instance"
-
-  protocol = "TCP"
-  port     = 6443
-
-  # TCP health check for apiserver
-  health_check {
-    protocol = "TCP"
-    port     = 6443
-
-    # NLBs required to use same healthy and unhealthy thresholds
-    healthy_threshold   = 3
-    unhealthy_threshold = 3
-
-    # Interval between health checks required to be 10 or 30
-    interval = 10
-  }
-}
-
-# Attach controller instances to apiserver NLB
-resource "aws_lb_target_group_attachment" "controllers" {
-  count = "${var.controller_count}"
-
-  target_group_arn = "${aws_lb_target_group.controllers.arn}"
-  target_id        = "${element(aws_instance.controllers.*.id, count.index)}"
-  port             = 6443
-}
--- a/aws/fedora-atomic/kubernetes/outputs.tf
+++ b/aws/fedora-atomic/kubernetes/outputs.tf
@ -1,48 +0,0 @@
-output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
-}
-
-# Outputs for Kubernetes Ingress
-
-output "ingress_dns_name" {
-  value       = "${aws_lb.nlb.dns_name}"
-  description = "DNS name of the network load balancer for distributing traffic to Ingress controllers"
-}
-
-output "ingress_zone_id" {
-  value       = "${aws_lb.nlb.zone_id}"
-  description = "Route53 zone id of the network load balancer DNS name that can be used in Route53 alias records"
-}
-
-# Outputs for worker pools
-
-output "vpc_id" {
-  value       = "${aws_vpc.network.id}"
-  description = "ID of the VPC for creating worker instances"
-}
-
-output "subnet_ids" {
-  value       = ["${aws_subnet.public.*.id}"]
-  description = "List of subnet IDs for creating worker instances"
-}
-
-output "worker_security_groups" {
-  value       = ["${aws_security_group.worker.id}"]
-  description = "List of worker security group IDs"
-}
-
-output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig-kubelet}"
-}
-
-# Outputs for custom load balancing
-
-output "worker_target_group_http" {
-  description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${module.workers.target_group_http}"
-}
-
-output "worker_target_group_https" {
-  description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${module.workers.target_group_https}"
-}
--- a/aws/fedora-atomic/kubernetes/require.tf
+++ b/aws/fedora-atomic/kubernetes/require.tf
@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "aws" {
-  version = "~> 1.13"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}
--- a/aws/fedora-atomic/kubernetes/security.tf
+++ b/aws/fedora-atomic/kubernetes/security.tf
@ -1,351 +0,0 @@
-# Security Groups (instance firewalls)
-
-# Controller security group
-
-resource "aws_security_group" "controller" {
-  name        = "${var.cluster_name}-controller"
-  description = "${var.cluster_name} controller security group"
-
-  vpc_id = "${aws_vpc.network.id}"
-
-  tags = "${map("Name", "${var.cluster_name}-controller")}"
-}
-
-resource "aws_security_group_rule" "controller-ssh" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 22
-  to_port     = 22
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "controller-etcd" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 2379
-  to_port   = 2380
-  self      = true
-}
-
-# Allow Prometheus to scrape etcd metrics
-resource "aws_security_group_rule" "controller-etcd-metrics" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 2381
-  to_port                  = 2381
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-apiserver" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 6443
-  to_port     = 6443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "controller-flannel" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "udp"
-  from_port                = 8472
-  to_port                  = 8472
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-flannel-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "udp"
-  from_port = 8472
-  to_port   = 8472
-  self      = true
-}
-
-# Allow Prometheus to scrape node-exporter daemonset
-resource "aws_security_group_rule" "controller-node-exporter" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 9100
-  to_port                  = 9100
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-# Allow apiserver to access kubelets for exec, log, port-forward
-resource "aws_security_group_rule" "controller-kubelet" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10250
-  to_port                  = 10250
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-kubelet-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10250
-  to_port   = 10250
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-bgp" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 179
-  to_port                  = 179
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-bgp-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 179
-  to_port   = 179
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-ipip" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = 4
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-ipip-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = 4
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-ipip-legacy" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type                     = "ingress"
-  protocol                 = 94
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.worker.id}"
-}
-
-resource "aws_security_group_rule" "controller-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type      = "ingress"
-  protocol  = 94
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "controller-egress" {
-  security_group_id = "${aws_security_group.controller.id}"
-
-  type             = "egress"
-  protocol         = "-1"
-  from_port        = 0
-  to_port          = 0
-  cidr_blocks      = ["0.0.0.0/0"]
-  ipv6_cidr_blocks = ["::/0"]
-}
-
-# Worker security group
-
-resource "aws_security_group" "worker" {
-  name        = "${var.cluster_name}-worker"
-  description = "${var.cluster_name} worker security group"
-
-  vpc_id = "${aws_vpc.network.id}"
-
-  tags = "${map("Name", "${var.cluster_name}-worker")}"
-}
-
-resource "aws_security_group_rule" "worker-ssh" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 22
-  to_port     = 22
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "worker-http" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 80
-  to_port     = 80
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "worker-https" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 443
-  to_port     = 443
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-resource "aws_security_group_rule" "worker-flannel" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "udp"
-  from_port                = 8472
-  to_port                  = 8472
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-flannel-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "udp"
-  from_port = 8472
-  to_port   = 8472
-  self      = true
-}
-
-# Allow Prometheus to scrape node-exporter daemonset
-resource "aws_security_group_rule" "worker-node-exporter" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 9100
-  to_port   = 9100
-  self      = true
-}
-
-resource "aws_security_group_rule" "ingress-health" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type        = "ingress"
-  protocol    = "tcp"
-  from_port   = 10254
-  to_port     = 10254
-  cidr_blocks = ["0.0.0.0/0"]
-}
-
-# Allow apiserver to access kubelets for exec, log, port-forward
-resource "aws_security_group_rule" "worker-kubelet" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 10250
-  to_port                  = 10250
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-# Allow Prometheus to scrape kubelet metrics
-resource "aws_security_group_rule" "worker-kubelet-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 10250
-  to_port   = 10250
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-bgp" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = "tcp"
-  from_port                = 179
-  to_port                  = 179
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-bgp-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = "tcp"
-  from_port = 179
-  to_port   = 179
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-ipip" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = 4
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-ipip-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = 4
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-ipip-legacy" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type                     = "ingress"
-  protocol                 = 94
-  from_port                = 0
-  to_port                  = 0
-  source_security_group_id = "${aws_security_group.controller.id}"
-}
-
-resource "aws_security_group_rule" "worker-ipip-legacy-self" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type      = "ingress"
-  protocol  = 94
-  from_port = 0
-  to_port   = 0
-  self      = true
-}
-
-resource "aws_security_group_rule" "worker-egress" {
-  security_group_id = "${aws_security_group.worker.id}"
-
-  type             = "egress"
-  protocol         = "-1"
-  from_port        = 0
-  to_port          = 0
-  cidr_blocks      = ["0.0.0.0/0"]
-  ipv6_cidr_blocks = ["::/0"]
-}
--- a/aws/fedora-atomic/kubernetes/ssh.tf
+++ b/aws/fedora-atomic/kubernetes/ssh.tf
@ -1,89 +0,0 @@
-# Secure copy etcd TLS assets to controllers.
-resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
-
-  connection {
-    type    = "ssh"
-    host    = "${element(aws_instance.controllers.*.public_ip, count.index)}"
-    user    = "fedora"
-    timeout = "15m"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
-    destination = "$HOME/etcd-client-ca.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
-    destination = "$HOME/etcd-client.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
-    destination = "$HOME/etcd-client.key"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
-    destination = "$HOME/etcd-server.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
-    destination = "$HOME/etcd-server.key"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
-    destination = "$HOME/etcd-peer.crt"
-  }
-
-  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
-    destination = "$HOME/etcd-peer.key"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "sudo mkdir -p /etc/ssl/etcd/etcd",
-      "sudo mv etcd-client* /etc/ssl/etcd/",
-      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/server-ca.crt",
-      "sudo mv etcd-server.crt /etc/ssl/etcd/etcd/server.crt",
-      "sudo mv etcd-server.key /etc/ssl/etcd/etcd/server.key",
-      "sudo cp /etc/ssl/etcd/etcd-client-ca.crt /etc/ssl/etcd/etcd/peer-ca.crt",
-      "sudo mv etcd-peer.crt /etc/ssl/etcd/etcd/peer.crt",
-      "sudo mv etcd-peer.key /etc/ssl/etcd/etcd/peer.key",
-    ]
-  }
-}
-
-# Secure copy bootkube assets to ONE controller and start bootkube to perform
-# one-time self-hosted cluster bootstrapping.
-resource "null_resource" "bootkube-start" {
-  depends_on = [
-    "null_resource.copy-controller-secrets",
-    "module.workers",
-    "aws_route53_record.apiserver",
-  ]
-
-  connection {
-    type    = "ssh"
-    host    = "${aws_instance.controllers.0.public_ip}"
-    user    = "fedora"
-    timeout = "15m"
-  }
-
-  provisioner "file" {
-    source      = "${var.asset_dir}"
-    destination = "$HOME/assets"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do sleep 4; done",
-      "sudo mv $HOME/assets /var/lib/bootkube",
-      "sudo systemctl start bootkube",
-    ]
-  }
-}
--- a/aws/fedora-atomic/kubernetes/variables.tf
+++ b/aws/fedora-atomic/kubernetes/variables.tf
@ -1,124 +0,0 @@
-variable "cluster_name" {
-  type        = "string"
-  description = "Unique cluster name (prepended to dns_zone)"
-}
-
-# AWS
-
-variable "dns_zone" {
-  type        = "string"
-  description = "AWS DNS Zone (e.g. aws.example.com)"
-}
-
-variable "dns_zone_id" {
-  type        = "string"
-  description = "AWS DNS Zone ID (e.g. Z3PAABBCFAKEC0)"
-}
-
-# instances
-
-variable "controller_count" {
-  type        = "string"
-  default     = "1"
-  description = "Number of controllers (i.e. masters)"
-}
-
-variable "worker_count" {
-  type        = "string"
-  default     = "1"
-  description = "Number of workers"
-}
-
-variable "controller_type" {
-  type        = "string"
-  default     = "t3.small"
-  description = "EC2 instance type for controllers"
-}
-
-variable "worker_type" {
-  type        = "string"
-  default     = "t3.small"
-  description = "EC2 instance type for workers"
-}
-
-variable "disk_size" {
-  type        = "string"
-  default     = "40"
-  description = "Size of the EBS volume in GB"
-}
-
-variable "disk_type" {
-  type        = "string"
-  default     = "gp2"
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-}
-
-variable "disk_iops" {
-  type        = "string"
-  default     = "0"
-  description = "IOPS of the EBS volume (e.g. 100)"
-}
-
-variable "worker_price" {
-  type        = "string"
-  default     = ""
-  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
-}
-
-# configuration
-
-variable "ssh_authorized_key" {
-  type        = "string"
-  description = "SSH public key for user 'fedora'"
-}
-
-variable "asset_dir" {
-  description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
-}
-
-variable "networking" {
-  description = "Choice of networking provider (calico or flannel)"
-  type        = "string"
-  default     = "calico"
-}
-
-variable "network_mtu" {
-  description = "CNI interface MTU (applies to calico only). Use 8981 if using instances types with Jumbo frames."
-  type        = "string"
-  default     = "1480"
-}
-
-variable "host_cidr" {
-  description = "CIDR IPv4 range to assign to EC2 nodes"
-  type        = "string"
-  default     = "10.0.0.0/16"
-}
-
-variable "pod_cidr" {
-  description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
-  default     = "10.2.0.0/16"
-}
-
-variable "service_cidr" {
-  description = <<EOD
-CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
-EOD
-
-  type    = "string"
-  default = "10.3.0.0/16"
-}
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
-}
-
-variable "enable_reporting" {
-  type        = "string"
-  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
-}
--- a/aws/fedora-atomic/kubernetes/workers.tf
+++ b/aws/fedora-atomic/kubernetes/workers.tf
@ -1,19 +0,0 @@
-module "workers" {
-  source = "workers"
-  name   = "${var.cluster_name}"
-
-  # AWS
-  vpc_id          = "${aws_vpc.network.id}"
-  subnet_ids      = ["${aws_subnet.public.*.id}"]
-  security_groups = ["${aws_security_group.worker.id}"]
-  count           = "${var.worker_count}"
-  instance_type   = "${var.worker_type}"
-  disk_size       = "${var.disk_size}"
-  spot_price      = "${var.worker_price}"
-
-  # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
-  ssh_authorized_key    = "${var.ssh_authorized_key}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-}
--- a/aws/fedora-atomic/kubernetes/workers/ami.tf
+++ b/aws/fedora-atomic/kubernetes/workers/ami.tf
@ -1,19 +0,0 @@
-data "aws_ami" "fedora" {
-  most_recent = true
-  owners      = ["125523088429"]
-
-  filter {
-    name   = "architecture"
-    values = ["x86_64"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "name"
-    values = ["Fedora-AtomicHost-28-20180625.1.x86_64-*-gp2-*"]
-  }
-}
--- a/aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl
+++ b/aws/fedora-atomic/kubernetes/workers/cloudinit/worker.yaml.tmpl
@ -1,66 +0,0 @@
-#cloud-config
-write_files:
-  - path: /etc/systemd/system/kubelet.service.d/10-typhoon.conf
-    content: |
-      [Unit]
-      Wants=rpc-statd.service
-      [Service]
-      ExecStartPre=/bin/mkdir -p /opt/cni/bin
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-      ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-      ExecStartPre=/bin/mkdir -p /var/lib/cni
-      ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-      ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-      Restart=always
-      RestartSec=10
-  - path: /etc/kubernetes/kubelet.conf
-    content: |
-      ARGS="--anonymous-auth=false \
-        --authentication-token-webhook \
-        --authorization-mode=Webhook \
-        --client-ca-file=/etc/kubernetes/ca.crt \
-        --cluster_dns=${cluster_dns_service_ip} \
-        --cluster_domain=${cluster_domain_suffix} \
-        --cni-conf-dir=/etc/kubernetes/cni/net.d \
-        --exit-on-lock-contention \
-        --kubeconfig=/etc/kubernetes/kubeconfig \
-        --lock-file=/var/run/lock/kubelet.lock \
-        --network-plugin=cni \
-        --node-labels=node-role.kubernetes.io/node \
-        --pod-manifest-path=/etc/kubernetes/manifests \
-        --read-only-port=0 \
-        --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
-  - path: /etc/kubernetes/kubeconfig
-    permissions: '0644'
-    content: |
-      ${kubeconfig}
-  - path: /etc/NetworkManager/conf.d/typhoon.conf
-    content: |
-      [main]
-      plugins=keyfile
-      [keyfile]
-      unmanaged-devices=interface-name:cali*;interface-name:tunl*
-  - path: /etc/selinux/config
-    owner: root:root
-    permissions: '0644'
-    content: |
-      SELINUX=permissive
-      SELINUXTYPE=targeted
-bootcmd:
-  - [setenforce, Permissive]
-  - [systemctl, disable, firewalld, --now]
-  # https://github.com/kubernetes/kubernetes/issues/60869
-  - [modprobe, ip_vs]
-runcmd:
-  - [systemctl, daemon-reload]
-  - [systemctl, restart, NetworkManager]
-  - "atomic install --system --name=kubelet quay.io/poseidon/kubelet:v1.13.4"
-  - [systemctl, start, --no-block, kubelet.service]
-users:
-  - default
-  - name: fedora
-    gecos: Fedora Admin
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: wheel,adm,systemd-journal,docker
-    ssh-authorized-keys:
-      - "${ssh_authorized_key}"
--- a/aws/fedora-atomic/kubernetes/workers/ingress.tf
+++ b/aws/fedora-atomic/kubernetes/workers/ingress.tf
@ -1,47 +0,0 @@
-# Target groups of instances for use with load balancers
-
-resource "aws_lb_target_group" "workers-http" {
-  name        = "${var.name}-workers-http"
-  vpc_id      = "${var.vpc_id}"
-  target_type = "instance"
-
-  protocol = "TCP"
-  port     = 80
-
-  # HTTP health check for ingress
-  health_check {
-    protocol = "HTTP"
-    port     = 10254
-    path     = "/healthz"
-
-    # NLBs required to use same healthy and unhealthy thresholds
-    healthy_threshold   = 3
-    unhealthy_threshold = 3
-
-    # Interval between health checks required to be 10 or 30
-    interval = 10
-  }
-}
-
-resource "aws_lb_target_group" "workers-https" {
-  name        = "${var.name}-workers-https"
-  vpc_id      = "${var.vpc_id}"
-  target_type = "instance"
-
-  protocol = "TCP"
-  port     = 443
-
-  # HTTP health check for ingress
-  health_check {
-    protocol = "HTTP"
-    port     = 10254
-    path     = "/healthz"
-
-    # NLBs required to use same healthy and unhealthy thresholds
-    healthy_threshold   = 3
-    unhealthy_threshold = 3
-
-    # Interval between health checks required to be 10 or 30
-    interval = 10
-  }
-}
--- a/aws/fedora-atomic/kubernetes/workers/outputs.tf
+++ b/aws/fedora-atomic/kubernetes/workers/outputs.tf
@ -1,9 +0,0 @@
-output "target_group_http" {
-  description = "ARN of a target group of workers for HTTP traffic"
-  value       = "${aws_lb_target_group.workers-http.arn}"
-}
-
-output "target_group_https" {
-  description = "ARN of a target group of workers for HTTPS traffic"
-  value       = "${aws_lb_target_group.workers-https.arn}"
-}
--- a/aws/fedora-atomic/kubernetes/workers/variables.tf
+++ b/aws/fedora-atomic/kubernetes/workers/variables.tf
@ -1,87 +0,0 @@
-variable "name" {
-  type        = "string"
-  description = "Unique name for the worker pool"
-}
-
-# AWS
-
-variable "vpc_id" {
-  type        = "string"
-  description = "Must be set to `vpc_id` output by cluster"
-}
-
-variable "subnet_ids" {
-  type        = "list"
-  description = "Must be set to `subnet_ids` output by cluster"
-}
-
-variable "security_groups" {
-  type        = "list"
-  description = "Must be set to `worker_security_groups` output by cluster"
-}
-
-# instances
-
-variable "count" {
-  type        = "string"
-  default     = "1"
-  description = "Number of instances"
-}
-
-variable "instance_type" {
-  type        = "string"
-  default     = "t3.small"
-  description = "EC2 instance type"
-}
-
-variable "disk_size" {
-  type        = "string"
-  default     = "40"
-  description = "Size of the EBS volume in GB"
-}
-
-variable "disk_type" {
-  type        = "string"
-  default     = "gp2"
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-}
-
-variable "disk_iops" {
-  type        = "string"
-  default     = "0"
-  description = "IOPS of the EBS volume (required for io1)"
-}
-
-variable "spot_price" {
-  type        = "string"
-  default     = ""
-  description = "Spot price in USD for autoscaling group spot instances. Leave as default empty string for autoscaling group to use on-demand instances. Note, switching in-place from spot to on-demand is not possible: https://github.com/terraform-providers/terraform-provider-aws/issues/4320"
-}
-
-# configuration
-
-variable "kubeconfig" {
-  type        = "string"
-  description = "Must be set to `kubeconfig` output by cluster"
-}
-
-variable "ssh_authorized_key" {
-  type        = "string"
-  description = "SSH public key for user 'fedora'"
-}
-
-variable "service_cidr" {
-  description = <<EOD
-CIDR IPv4 range to assign Kubernetes services.
-The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
-EOD
-
-  type    = "string"
-  default = "10.3.0.0/16"
-}
-
-variable "cluster_domain_suffix" {
-  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
-}
--- a/aws/fedora-atomic/kubernetes/workers/workers.tf
+++ b/aws/fedora-atomic/kubernetes/workers/workers.tf
@ -1,78 +0,0 @@
-# Workers AutoScaling Group
-resource "aws_autoscaling_group" "workers" {
-  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
-
-  # count
-  desired_capacity          = "${var.count}"
-  min_size                  = "${var.count}"
-  max_size                  = "${var.count + 2}"
-  default_cooldown          = 30
-  health_check_grace_period = 30
-
-  # network
-  vpc_zone_identifier = ["${var.subnet_ids}"]
-
-  # template
-  launch_configuration = "${aws_launch_configuration.worker.name}"
-
-  # target groups to which instances should be added
-  target_group_arns = [
-    "${aws_lb_target_group.workers-http.id}",
-    "${aws_lb_target_group.workers-https.id}",
-  ]
-
-  lifecycle {
-    # override the default destroy and replace update behavior
-    create_before_destroy = true
-  }
-
-  # Waiting for instance creation delays adding the ASG to state. If instances
-  # can't be created (e.g. spot price too low), the ASG will be orphaned.
-  # Orphaned ASGs escape cleanup, can't be updated, and keep bidding if spot is
-  # used. Disable wait to avoid issues and align with other clouds.
-  wait_for_capacity_timeout = "0"
-
-  tags = [{
-    key                 = "Name"
-    value               = "${var.name}-worker"
-    propagate_at_launch = true
-  }]
-}
-
-# Worker template
-resource "aws_launch_configuration" "worker" {
-  image_id          = "${data.aws_ami.fedora.image_id}"
-  instance_type     = "${var.instance_type}"
-  spot_price        = "${var.spot_price}"
-  enable_monitoring = false
-
-  user_data = "${data.template_file.worker-cloudinit.rendered}"
-
-  # storage
-  root_block_device {
-    volume_type = "${var.disk_type}"
-    volume_size = "${var.disk_size}"
-    iops        = "${var.disk_iops}"
-  }
-
-  # network
-  security_groups = ["${var.security_groups}"]
-
-  lifecycle {
-    // Override the default destroy and replace update behavior
-    create_before_destroy = true
-    ignore_changes        = ["image_id"]
-  }
-}
-
-# Worker Cloud-Init
-data "template_file" "worker-cloudinit" {
-  template = "${file("${path.module}/cloudinit/worker.yaml.tmpl")}"
-
-  vars = {
-    kubeconfig             = "${indent(6, var.kubeconfig)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-  }
-}
--- a/aws/ignore/.gitkeep
+++ b/aws/ignore/.gitkeep
--- a/azure/container-linux/kubernetes/README.md
+++ b/azure/container-linux/kubernetes/README.md
@ -11,9 +11,9 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
-* Single or multi-master, [flannel](https://github.com/coreos/flannel) networking
-* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
+* On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [low-priority](https://typhoon.psdn.io/cl/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

--- a/azure/container-linux/kubernetes/bootkube.tf
+++ b/azure/container-linux/kubernetes/bootkube.tf
@ -1,14 +1,23 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"

-  cluster_name          = "${var.cluster_name}"
-  api_servers           = ["${format("%s.%s", var.cluster_name, var.dns_zone)}"]
-  etcd_servers          = ["${formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)}"]
-  asset_dir             = "${var.asset_dir}"
-  networking            = "flannel"
-  pod_cidr              = "${var.pod_cidr}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  enable_reporting      = "${var.enable_reporting}"
+  cluster_name = var.cluster_name
+  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
+  etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
+  asset_dir    = var.asset_dir
+
+  networking = var.networking
+
+  # only effective with Calico networking
+  # we should be able to use 1450 MTU, but in practice, 1410 was needed
+  network_encapsulation = "vxlan"
+  network_mtu           = "1410"
+
+  pod_cidr              = var.pod_cidr
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  enable_reporting      = var.enable_reporting
+  enable_aggregation    = var.enable_aggregation
 }
+
--- a/azure/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/azure/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.12"
+            Environment="ETCD_IMAGE_TAG=v3.3.13"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
@ -123,7 +123,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.4
+          KUBELET_IMAGE_TAG=v1.15.0
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
--- a/azure/container-linux/kubernetes/controllers.tf
+++ b/azure/container-linux/kubernetes/controllers.tf
@ -1,31 +1,34 @@
 # Discrete DNS records for each controller's private IPv4 for etcd usage
 resource "azurerm_dns_a_record" "etcds" {
-  count               = "${var.controller_count}"
-  resource_group_name = "${var.dns_zone_group}"
+  count               = var.controller_count
+  resource_group_name = var.dns_zone_group

  # DNS Zone name where record should be created
-  zone_name = "${var.dns_zone}"
+  zone_name = var.dns_zone

  # DNS record
-  name = "${format("%s-etcd%d", var.cluster_name, count.index)}"
+  name = format("%s-etcd%d", var.cluster_name, count.index)
  ttl  = 300

  # private IPv4 address for etcd
-  records = ["${element(azurerm_network_interface.controllers.*.private_ip_address, count.index)}"]
+  records = [element(
+    azurerm_network_interface.controllers.*.private_ip_address,
+    count.index,
+  )]
 }

 locals {
  # Channel for a Container Linux derivative
  # coreos-stable -> Container Linux Stable
-  channel = "${element(split("-", var.os_image), 1)}"
+  channel = element(split("-", var.os_image), 1)
 }

 # Controller availability set to spread controllers
 resource "azurerm_availability_set" "controllers" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                         = "${var.cluster_name}-controllers"
-  location                     = "${var.region}"
+  location                     = var.region
  platform_fault_domain_count  = 2
  platform_update_domain_count = 4
  managed                      = true
@ -33,19 +36,19 @@ resource "azurerm_availability_set" "controllers" {

 # Controller instances
 resource "azurerm_virtual_machine" "controllers" {
-  count               = "${var.controller_count}"
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  count               = var.controller_count
+  resource_group_name = azurerm_resource_group.cluster.name

  name                = "${var.cluster_name}-controller-${count.index}"
-  location            = "${var.region}"
-  availability_set_id = "${azurerm_availability_set.controllers.id}"
-  vm_size             = "${var.controller_type}"
+  location            = var.region
+  availability_set_id = azurerm_availability_set.controllers.id
+  vm_size             = var.controller_type

  # boot
  storage_image_reference {
    publisher = "CoreOS"
    offer     = "CoreOS"
-    sku       = "${local.channel}"
+    sku       = local.channel
    version   = "latest"
  }

@ -54,18 +57,18 @@ resource "azurerm_virtual_machine" "controllers" {
    name              = "${var.cluster_name}-controller-${count.index}"
    create_option     = "FromImage"
    caching           = "ReadWrite"
-    disk_size_gb      = "${var.disk_size}"
+    disk_size_gb      = var.disk_size
    os_type           = "Linux"
    managed_disk_type = "Premium_LRS"
  }

  # network
-  network_interface_ids = ["${element(azurerm_network_interface.controllers.*.id, count.index)}"]
+  network_interface_ids = [element(azurerm_network_interface.controllers.*.id, count.index)]

  os_profile {
    computer_name  = "${var.cluster_name}-controller-${count.index}"
    admin_username = "core"
-    custom_data    = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+    custom_data    = element(data.ct_config.controller-ignitions.*.rendered, count.index)
  }

  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@ -74,7 +77,7 @@ resource "azurerm_virtual_machine" "controllers" {

    ssh_keys {
      path     = "/home/core/.ssh/authorized_keys"
-      key_data = "${var.ssh_authorized_key}"
+      key_data = var.ssh_authorized_key
    }
  }

@ -84,85 +87,87 @@ resource "azurerm_virtual_machine" "controllers" {

  lifecycle {
    ignore_changes = [
-      "storage_os_disk",
-      "os_profile",
+      storage_os_disk,
+      os_profile,
    ]
  }
 }

 # Controller NICs with public and private IPv4
 resource "azurerm_network_interface" "controllers" {
-  count               = "${var.controller_count}"
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  count               = var.controller_count
+  resource_group_name = azurerm_resource_group.cluster.name

  name                      = "${var.cluster_name}-controller-${count.index}"
-  location                  = "${azurerm_resource_group.cluster.location}"
-  network_security_group_id = "${azurerm_network_security_group.controller.id}"
+  location                  = azurerm_resource_group.cluster.location
+  network_security_group_id = azurerm_network_security_group.controller.id

  ip_configuration {
    name                          = "ip0"
-    subnet_id                     = "${azurerm_subnet.controller.id}"
+    subnet_id                     = azurerm_subnet.controller.id
    private_ip_address_allocation = "dynamic"

    # public IPv4
-    public_ip_address_id = "${element(azurerm_public_ip.controllers.*.id, count.index)}"
+    public_ip_address_id = element(azurerm_public_ip.controllers.*.id, count.index)
  }
 }

 # Add controller NICs to the controller backend address pool
 resource "azurerm_network_interface_backend_address_pool_association" "controllers" {
-  network_interface_id    = "${azurerm_network_interface.controllers.id}"
+  network_interface_id    = azurerm_network_interface.controllers[0].id
  ip_configuration_name   = "ip0"
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
 }

 # Controller public IPv4 addresses
 resource "azurerm_public_ip" "controllers" {
-  count               = "${var.controller_count}"
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  count               = var.controller_count
+  resource_group_name = azurerm_resource_group.cluster.name

  name              = "${var.cluster_name}-controller-${count.index}"
-  location          = "${azurerm_resource_group.cluster.location}"
+  location          = azurerm_resource_group.cluster.location
  sku               = "Standard"
  allocation_method = "Static"
 }

 # Controller Ignition configs
 data "ct_config" "controller-ignitions" {
-  count        = "${var.controller_count}"
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  count = var.controller_count
+  content = element(
+    data.template_file.controller-configs.*.rendered,
+    count.index,
+  )
  pretty_print = false
-  snippets     = ["${var.controller_clc_snippets}"]
+  snippets     = var.controller_clc_snippets
 }

 # Controller Container Linux configs
 data "template_file" "controller-configs" {
-  count = "${var.controller_count}"
+  count = var.controller_count

-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")

  vars = {
    # Cannot use cyclic dependencies on controllers or their DNS records
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
-
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster = "${join(",", data.template_file.etcds.*.rendered)}"
-
-    kubeconfig             = "${indent(10, module.bootkube.kubeconfig-kubelet)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    kubeconfig             = indent(10, module.bootkube.kubeconfig-kubelet)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
  }
 }

 data "template_file" "etcds" {
-  count    = "${var.controller_count}"
+  count    = var.controller_count
  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"

-  vars {
-    index        = "${count.index}"
-    cluster_name = "${var.cluster_name}"
-    dns_zone     = "${var.dns_zone}"
+  vars = {
+    index        = count.index
+    cluster_name = var.cluster_name
+    dns_zone     = var.dns_zone
  }
 }
+
--- a/azure/container-linux/kubernetes/lb.tf
+++ b/azure/container-linux/kubernetes/lb.tf
@ -1,123 +1,123 @@
 # DNS record for the apiserver load balancer
 resource "azurerm_dns_a_record" "apiserver" {
-  resource_group_name = "${var.dns_zone_group}"
+  resource_group_name = var.dns_zone_group

  # DNS Zone name where record should be created
-  zone_name = "${var.dns_zone}"
+  zone_name = var.dns_zone

  # DNS record
-  name = "${var.cluster_name}"
+  name = var.cluster_name
  ttl  = 300

  # IPv4 address of apiserver load balancer
-  records = ["${azurerm_public_ip.apiserver-ipv4.ip_address}"]
+  records = [azurerm_public_ip.apiserver-ipv4.ip_address]
 }

 # Static IPv4 address for the apiserver frontend
 resource "azurerm_public_ip" "apiserver-ipv4" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name              = "${var.cluster_name}-apiserver-ipv4"
-  location          = "${var.region}"
+  location          = var.region
  sku               = "Standard"
  allocation_method = "Static"
 }

 # Static IPv4 address for the ingress frontend
 resource "azurerm_public_ip" "ingress-ipv4" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name              = "${var.cluster_name}-ingress-ipv4"
-  location          = "${var.region}"
+  location          = var.region
  sku               = "Standard"
  allocation_method = "Static"
 }

 # Network Load Balancer for apiservers and ingress
 resource "azurerm_lb" "cluster" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

-  name     = "${var.cluster_name}"
-  location = "${var.region}"
+  name     = var.cluster_name
+  location = var.region
  sku      = "Standard"

  frontend_ip_configuration {
    name                 = "apiserver"
-    public_ip_address_id = "${azurerm_public_ip.apiserver-ipv4.id}"
+    public_ip_address_id = azurerm_public_ip.apiserver-ipv4.id
  }

  frontend_ip_configuration {
    name                 = "ingress"
-    public_ip_address_id = "${azurerm_public_ip.ingress-ipv4.id}"
+    public_ip_address_id = azurerm_public_ip.ingress-ipv4.id
  }
 }

 resource "azurerm_lb_rule" "apiserver" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                           = "apiserver"
-  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

  protocol                = "Tcp"
  frontend_port           = 6443
  backend_port            = 6443
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.controller.id}"
-  probe_id                = "${azurerm_lb_probe.apiserver.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
+  probe_id                = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                           = "ingress-http"
-  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "ingress"

  protocol                = "Tcp"
  frontend_port           = 80
  backend_port            = 80
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
-  probe_id                = "${azurerm_lb_probe.ingress.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  probe_id                = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                           = "ingress-https"
-  loadbalancer_id                = "${azurerm_lb.cluster.id}"
+  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "ingress"

  protocol                = "Tcp"
  frontend_port           = 443
  backend_port            = 443
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
-  probe_id                = "${azurerm_lb_probe.ingress.id}"
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
+  probe_id                = azurerm_lb_probe.ingress.id
 }

 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name            = "controller"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
 }

 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name            = "worker"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
 }

 # Health checks / probes

 # TCP health check for apiserver
 resource "azurerm_lb_probe" "apiserver" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name            = "apiserver"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
  protocol        = "Tcp"
  port            = 6443

@ -129,10 +129,10 @@ resource "azurerm_lb_probe" "apiserver" {

 # HTTP health check for ingress
 resource "azurerm_lb_probe" "ingress" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name            = "ingress"
-  loadbalancer_id = "${azurerm_lb.cluster.id}"
+  loadbalancer_id = azurerm_lb.cluster.id
  protocol        = "Http"
  port            = 10254
  request_path    = "/healthz"
@ -142,3 +142,4 @@ resource "azurerm_lb_probe" "ingress" {

  interval_in_seconds = 5
 }
+
--- a/azure/container-linux/kubernetes/network.tf
+++ b/azure/container-linux/kubernetes/network.tf
@ -1,15 +1,15 @@
 # Organize cluster into a resource group
 resource "azurerm_resource_group" "cluster" {
-  name     = "${var.cluster_name}"
-  location = "${var.region}"
+  name     = var.cluster_name
+  location = var.region
 }

 resource "azurerm_virtual_network" "network" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

-  name          = "${var.cluster_name}"
-  location      = "${azurerm_resource_group.cluster.location}"
-  address_space = ["${var.host_cidr}"]
+  name          = var.cluster_name
+  location      = azurerm_resource_group.cluster.location
+  address_space = [var.host_cidr]
 }

 # Subnets - separate subnets for controller and workers because Azure
@ -17,17 +17,18 @@ resource "azurerm_virtual_network" "network" {
 # tags like GCP or security group membership like AWS

 resource "azurerm_subnet" "controller" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                 = "controller"
-  virtual_network_name = "${azurerm_virtual_network.network.name}"
-  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 0)}"
+  virtual_network_name = azurerm_virtual_network.network.name
+  address_prefix       = cidrsubnet(var.host_cidr, 1, 0)
 }

 resource "azurerm_subnet" "worker" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                 = "worker"
-  virtual_network_name = "${azurerm_virtual_network.network.name}"
-  address_prefix       = "${cidrsubnet(var.host_cidr, 1, 1)}"
+  virtual_network_name = azurerm_virtual_network.network.name
+  address_prefix       = cidrsubnet(var.host_cidr, 1, 1)
 }
+
--- a/azure/container-linux/kubernetes/outputs.tf
+++ b/azure/container-linux/kubernetes/outputs.tf
@ -1,36 +1,56 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }

 # Outputs for Kubernetes Ingress

 output "ingress_static_ipv4" {
-  value       = "${azurerm_public_ip.ingress-ipv4.ip_address}"
+  value       = azurerm_public_ip.ingress-ipv4.ip_address
  description = "IPv4 address of the load balancer for distributing traffic to Ingress controllers"
 }

 # Outputs for worker pools

 output "region" {
-  value = "${azurerm_resource_group.cluster.location}"
+  value = azurerm_resource_group.cluster.location
 }

 output "resource_group_name" {
-  value = "${azurerm_resource_group.cluster.name}"
+  value = azurerm_resource_group.cluster.name
 }

 output "subnet_id" {
-  value = "${azurerm_subnet.worker.id}"
+  value = azurerm_subnet.worker.id
 }

 output "security_group_id" {
-  value = "${azurerm_network_security_group.worker.id}"
-}
-
-output "backend_address_pool_id" {
-  value = "${azurerm_lb_backend_address_pool.worker.id}"
+  value = azurerm_network_security_group.worker.id
 }

 output "kubeconfig" {
-  value = "${module.bootkube.kubeconfig-kubelet}"
+  value = module.bootkube.kubeconfig-kubelet
 }
+
+# Outputs for custom firewalling
+
+output "worker_security_group_name" {
+  value = azurerm_network_security_group.worker.name
+}
+
+output "worker_address_prefix" {
+  description = "Worker network subnet CIDR address (for source/destination)"
+  value       = azurerm_subnet.worker.address_prefix
+}
+
+# Outputs for custom load balancing
+
+output "loadbalancer_id" {
+  description = "ID of the cluster load balancer"
+  value       = azurerm_lb.cluster.id
+}
+
+output "backend_address_pool_id" {
+  description = "ID of the worker backend address pool"
+  value       = azurerm_lb_backend_address_pool.worker.id
+}
+
--- a/azure/container-linux/kubernetes/require.tf
+++ b/azure/container-linux/kubernetes/require.tf
@ -1,25 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "azurerm" {
-  version = "~> 1.21"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}
--- a/azure/container-linux/kubernetes/security.tf
+++ b/azure/container-linux/kubernetes/security.tf
@ -1,17 +1,17 @@
 # Controller security group

 resource "azurerm_network_security_group" "controller" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name     = "${var.cluster_name}-controller"
-  location = "${azurerm_resource_group.cluster.location}"
+  location = azurerm_resource_group.cluster.location
 }

 resource "azurerm_network_security_rule" "controller-ssh" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-ssh"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2000"
  access                      = "Allow"
  direction                   = "Inbound"
@ -19,45 +19,45 @@ resource "azurerm_network_security_rule" "controller-ssh" {
  source_port_range           = "*"
  destination_port_range      = "22"
  source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

 resource "azurerm_network_security_rule" "controller-etcd" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-etcd"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2005"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "2379-2380"
-  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  source_address_prefix       = azurerm_subnet.controller.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

 # Allow Prometheus to scrape etcd metrics
 resource "azurerm_network_security_rule" "controller-etcd-metrics" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-etcd-metrics"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2010"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "2381"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

 resource "azurerm_network_security_rule" "controller-apiserver" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-apiserver"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2015"
  access                      = "Allow"
  direction                   = "Inbound"
@ -65,46 +65,46 @@ resource "azurerm_network_security_rule" "controller-apiserver" {
  source_port_range           = "*"
  destination_port_range      = "6443"
  source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

-resource "azurerm_network_security_rule" "controller-flannel" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+resource "azurerm_network_security_rule" "controller-vxlan" {
+  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-flannel"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  name                        = "allow-vxlan"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2020"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Udp"
  source_port_range           = "*"
-  destination_port_range      = "8472"
-  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  destination_port_range      = "4789"
+  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "controller-node-exporter" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-node-exporter"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2025"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "9100"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.controller.address_prefix}"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.controller.address_prefix
 }

 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "controller-kubelet" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-kubelet"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "2030"
  access                      = "Allow"
  direction                   = "Inbound"
@ -113,18 +113,18 @@ resource "azurerm_network_security_rule" "controller-kubelet" {
  destination_port_range      = "10250"

  # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
-  destination_address_prefix = "${azurerm_subnet.controller.address_prefix}"
+  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix = azurerm_subnet.controller.address_prefix
 }

 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules

 resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-loadbalancer"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "3000"
  access                      = "Allow"
  direction                   = "Inbound"
@ -136,10 +136,10 @@ resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
 }

 resource "azurerm_network_security_rule" "controller-deny-all" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "deny-all"
-  network_security_group_name = "${azurerm_network_security_group.controller.name}"
+  network_security_group_name = azurerm_network_security_group.controller.name
  priority                    = "3005"
  access                      = "Deny"
  direction                   = "Inbound"
@ -153,32 +153,32 @@ resource "azurerm_network_security_rule" "controller-deny-all" {
 # Worker security group

 resource "azurerm_network_security_group" "worker" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name     = "${var.cluster_name}-worker"
-  location = "${azurerm_resource_group.cluster.location}"
+  location = azurerm_resource_group.cluster.location
 }

 resource "azurerm_network_security_rule" "worker-ssh" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-ssh"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "2000"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "22"
-  source_address_prefix       = "${azurerm_subnet.controller.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  source_address_prefix       = azurerm_subnet.controller.address_prefix
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }

 resource "azurerm_network_security_rule" "worker-http" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-http"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "2005"
  access                      = "Allow"
  direction                   = "Inbound"
@ -186,14 +186,14 @@ resource "azurerm_network_security_rule" "worker-http" {
  source_port_range           = "*"
  destination_port_range      = "80"
  source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }

 resource "azurerm_network_security_rule" "worker-https" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-https"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "2010"
  access                      = "Allow"
  direction                   = "Inbound"
@ -201,46 +201,46 @@ resource "azurerm_network_security_rule" "worker-https" {
  source_port_range           = "*"
  destination_port_range      = "443"
  source_address_prefix       = "*"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }

-resource "azurerm_network_security_rule" "worker-flannel" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+resource "azurerm_network_security_rule" "worker-vxlan" {
+  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-flannel"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  name                        = "allow-vxlan"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "2015"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Udp"
  source_port_range           = "*"
-  destination_port_range      = "8472"
-  source_address_prefixes     = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  destination_port_range      = "4789"
+  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }

 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "worker-node-exporter" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-node-exporter"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "2020"
  access                      = "Allow"
  direction                   = "Inbound"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "9100"
-  source_address_prefix       = "${azurerm_subnet.worker.address_prefix}"
-  destination_address_prefix  = "${azurerm_subnet.worker.address_prefix}"
+  source_address_prefix       = azurerm_subnet.worker.address_prefix
+  destination_address_prefix  = azurerm_subnet.worker.address_prefix
 }

 # Allow apiserver to access kubelet's for exec, log, port-forward
 resource "azurerm_network_security_rule" "worker-kubelet" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-kubelet"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "2025"
  access                      = "Allow"
  direction                   = "Inbound"
@ -249,18 +249,18 @@ resource "azurerm_network_security_rule" "worker-kubelet" {
  destination_port_range      = "10250"

  # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = ["${azurerm_subnet.controller.address_prefix}", "${azurerm_subnet.worker.address_prefix}"]
-  destination_address_prefix = "${azurerm_subnet.worker.address_prefix}"
+  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
+  destination_address_prefix = azurerm_subnet.worker.address_prefix
 }

 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
 # https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules

 resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "allow-loadbalancer"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "3000"
  access                      = "Allow"
  direction                   = "Inbound"
@ -272,10 +272,10 @@ resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
 }

 resource "azurerm_network_security_rule" "worker-deny-all" {
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  resource_group_name = azurerm_resource_group.cluster.name

  name                        = "deny-all"
-  network_security_group_name = "${azurerm_network_security_group.worker.name}"
+  network_security_group_name = azurerm_network_security_group.worker.name
  priority                    = "3005"
  access                      = "Deny"
  direction                   = "Inbound"
@ -285,3 +285,4 @@ resource "azurerm_network_security_rule" "worker-deny-all" {
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
 }
+
--- a/azure/container-linux/kubernetes/ssh.tf
+++ b/azure/container-linux/kubernetes/ssh.tf
@ -1,50 +1,48 @@
 # Secure copy etcd TLS assets to controllers.
 resource "null_resource" "copy-controller-secrets" {
-  count = "${var.controller_count}"
+  count = var.controller_count

-  depends_on = [
-    "azurerm_virtual_machine.controllers",
-  ]
+  depends_on = [azurerm_virtual_machine.controllers]

  connection {
    type    = "ssh"
-    host    = "${element(azurerm_public_ip.controllers.*.ip_address, count.index)}"
+    host    = element(azurerm_public_ip.controllers.*.ip_address, count.index)
    user    = "core"
    timeout = "15m"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
    destination = "$HOME/etcd-client-ca.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
    destination = "$HOME/etcd-client.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
    destination = "$HOME/etcd-client.key"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
    destination = "$HOME/etcd-server.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
    destination = "$HOME/etcd-server.key"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
    destination = "$HOME/etcd-peer.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
    destination = "$HOME/etcd-peer.key"
  }

@ -68,21 +66,21 @@ resource "null_resource" "copy-controller-secrets" {
 # one-time self-hosted cluster bootstrapping.
 resource "null_resource" "bootkube-start" {
  depends_on = [
-    "module.bootkube",
-    "module.workers",
-    "azurerm_dns_a_record.apiserver",
-    "null_resource.copy-controller-secrets",
+    module.bootkube,
+    module.workers,
+    azurerm_dns_a_record.apiserver,
+    null_resource.copy-controller-secrets,
  ]

  connection {
    type    = "ssh"
-    host    = "${element(azurerm_public_ip.controllers.*.ip_address, 0)}"
+    host    = element(azurerm_public_ip.controllers.*.ip_address, 0)
    user    = "core"
    timeout = "15m"
  }

  provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
    destination = "$HOME/assets"
  }

@ -93,3 +91,4 @@ resource "null_resource" "bootkube-start" {
    ]
  }
 }
+
--- a/azure/container-linux/kubernetes/variables.tf
+++ b/azure/container-linux/kubernetes/variables.tf
@ -1,77 +1,77 @@
 variable "cluster_name" {
-  type        = "string"
+  type        = string
  description = "Unique cluster name (prepended to dns_zone)"
 }

 # Azure

 variable "region" {
-  type        = "string"
+  type        = string
  description = "Azure Region (e.g. centralus , see `az account list-locations --output table`)"
 }

 variable "dns_zone" {
-  type        = "string"
+  type        = string
  description = "Azure DNS Zone (e.g. azure.example.com)"
 }

 variable "dns_zone_group" {
-  type        = "string"
+  type        = string
  description = "Resource group where the Azure DNS Zone resides (e.g. global)"
 }

 # instances

 variable "controller_count" {
-  type        = "string"
+  type        = string
  default     = "1"
  description = "Number of controllers (i.e. masters)"
 }

 variable "worker_count" {
-  type        = "string"
+  type        = string
  default     = "1"
  description = "Number of workers"
 }

 variable "controller_type" {
-  type        = "string"
+  type        = string
  default     = "Standard_DS1_v2"
  description = "Machine type for controllers (see `az vm list-skus --location centralus`)"
 }

 variable "worker_type" {
-  type        = "string"
+  type        = string
  default     = "Standard_F1"
  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
 }

 variable "os_image" {
-  type        = "string"
+  type        = string
  default     = "coreos-stable"
  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
 }

 variable "disk_size" {
-  type        = "string"
+  type        = string
  default     = "40"
  description = "Size of the disk in GB"
 }

 variable "worker_priority" {
-  type        = "string"
+  type        = string
  default     = "Regular"
  description = "Set worker priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be deallocated at any time."
 }

 variable "controller_clc_snippets" {
-  type        = "list"
+  type        = list(string)
  description = "Controller Container Linux Config snippets"
  default     = []
 }

 variable "worker_clc_snippets" {
-  type        = "list"
+  type        = list(string)
  description = "Worker Container Linux Config snippets"
  default     = []
 }
@ -79,24 +79,30 @@ variable "worker_clc_snippets" {
 # configuration

 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
  description = "SSH public key for user 'core'"
 }

 variable "asset_dir" {
  description = "Path to a directory where generated assets should be placed (contains secrets)"
-  type        = "string"
+  type        = string
+}
+
+variable "networking" {
+  description = "Choice of networking provider (flannel or calico)"
+  type        = string
+  default     = "flannel"
 }

 variable "host_cidr" {
  description = "CIDR IPv4 range to assign to instances"
-  type        = "string"
+  type        = string
  default     = "10.0.0.0/16"
 }

 variable "pod_cidr" {
  description = "CIDR IPv4 range to assign Kubernetes pods"
-  type        = "string"
+  type        = string
  default     = "10.2.0.0/16"
 }

@ -106,18 +112,26 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD

-  type    = "string"
+
+  type = string
  default = "10.3.0.0/16"
 }

 variable "cluster_domain_suffix" {
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
+  type = string
+  default = "cluster.local"
 }

 variable "enable_reporting" {
-  type        = "string"
+  type = string
  description = "Enable usage or analytics reporting to upstreams (Calico)"
-  default     = "false"
+  default = "false"
 }
+
+variable "enable_aggregation" {
+  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
+  type = string
+  default = "false"
+}
+
--- a/azure/container-linux/kubernetes/versions.tf
+++ b/azure/container-linux/kubernetes/versions.tf
@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = "~> 0.12.0"
+  required_providers {
+    azurerm  = "~> 1.27"
+    ct       = "~> 0.3.2"
+    template = "~> 2.1"
+    null     = "~> 2.1"
+  }
+}
+
--- a/azure/container-linux/kubernetes/workers.tf
+++ b/azure/container-linux/kubernetes/workers.tf
@ -1,23 +1,24 @@
 module "workers" {
-  source = "workers"
-  name   = "${var.cluster_name}"
+  source = "./workers"
+  name   = var.cluster_name

  # Azure
-  resource_group_name     = "${azurerm_resource_group.cluster.name}"
-  region                  = "${azurerm_resource_group.cluster.location}"
-  subnet_id               = "${azurerm_subnet.worker.id}"
-  security_group_id       = "${azurerm_network_security_group.worker.id}"
-  backend_address_pool_id = "${azurerm_lb_backend_address_pool.worker.id}"
+  resource_group_name     = azurerm_resource_group.cluster.name
+  region                  = azurerm_resource_group.cluster.location
+  subnet_id               = azurerm_subnet.worker.id
+  security_group_id       = azurerm_network_security_group.worker.id
+  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id

-  count    = "${var.worker_count}"
-  vm_type  = "${var.worker_type}"
-  os_image = "${var.os_image}"
-  priority = "${var.worker_priority}"
+  worker_count = var.worker_count
+  vm_type      = var.worker_type
+  os_image     = var.os_image
+  priority     = var.worker_priority

  # configuration
-  kubeconfig            = "${module.bootkube.kubeconfig-kubelet}"
-  ssh_authorized_key    = "${var.ssh_authorized_key}"
-  service_cidr          = "${var.service_cidr}"
-  cluster_domain_suffix = "${var.cluster_domain_suffix}"
-  clc_snippets          = "${var.worker_clc_snippets}"
+  kubeconfig            = module.bootkube.kubeconfig-kubelet
+  ssh_authorized_key    = var.ssh_authorized_key
+  service_cidr          = var.service_cidr
+  cluster_domain_suffix = var.cluster_domain_suffix
+  clc_snippets          = var.worker_clc_snippets
 }
+
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml.tmpl
@ -93,7 +93,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.4
+          KUBELET_IMAGE_TAG=v1.15.0
    - path: /etc/sysctl.d/max-user-watches.conf
      filesystem: root
      contents:
@ -111,7 +111,7 @@ storage:
            --volume config,kind=host,source=/etc/kubernetes \
            --mount volume=config,target=/etc/kubernetes \
            --insecure-options=image \
-            docker://k8s.gcr.io/hyperkube:v1.13.4 \
+            docker://k8s.gcr.io/hyperkube:v1.15.0 \
            --net=host \
            --dns=host \
            --exec=/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
--- a/azure/container-linux/kubernetes/workers/variables.tf
+++ b/azure/container-linux/kubernetes/workers/variables.tf
@ -1,63 +1,63 @@
 variable "name" {
-  type        = "string"
+  type        = string
  description = "Unique name for the worker pool"
 }

 # Azure

 variable "region" {
-  type        = "string"
+  type        = string
  description = "Must be set to the Azure Region of cluster"
 }

 variable "resource_group_name" {
-  type        = "string"
+  type        = string
  description = "Must be set to the resource group name of cluster"
 }

 variable "subnet_id" {
-  type        = "string"
+  type        = string
  description = "Must be set to the `worker_subnet_id` output by cluster"
 }

 variable "security_group_id" {
-  type        = "string"
+  type        = string
  description = "Must be set to the `worker_security_group_id` output by cluster"
 }

 variable "backend_address_pool_id" {
-  type        = "string"
+  type        = string
  description = "Must be set to the `worker_backend_address_pool_id` output by cluster"
 }

 # instances

-variable "count" {
-  type        = "string"
+variable "worker_count" {
+  type        = string
  default     = "1"
  description = "Number of instances"
 }

 variable "vm_type" {
-  type        = "string"
+  type        = string
  default     = "Standard_F1"
  description = "Machine type for instances (see `az vm list-skus --location centralus`)"
 }

 variable "os_image" {
-  type        = "string"
+  type        = string
  default     = "coreos-stable"
  description = "Channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha)"
 }

 variable "priority" {
-  type        = "string"
+  type        = string
  default     = "Regular"
  description = "Set priority to Low to use reduced cost surplus capacity, with the tradeoff that instances can be evicted at any time."
 }

 variable "clc_snippets" {
-  type        = "list"
+  type        = list(string)
  description = "Container Linux Config snippets"
  default     = []
 }
@ -65,12 +65,12 @@ variable "clc_snippets" {
 # configuration

 variable "kubeconfig" {
-  type        = "string"
+  type        = string
  description = "Must be set to `kubeconfig` output by cluster"
 }

 variable "ssh_authorized_key" {
-  type        = "string"
+  type        = string
  description = "SSH public key for user 'core'"
 }

@ -80,12 +80,14 @@ CIDR IPv4 range to assign Kubernetes services.
 The 1st IP will be reserved for kube_apiserver, the 10th IP will be reserved for coredns.
 EOD

-  type    = "string"
+
+  type = string
  default = "10.3.0.0/16"
 }

 variable "cluster_domain_suffix" {
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
-  type        = "string"
-  default     = "cluster.local"
+  type = string
+  default = "cluster.local"
 }
+
--- a/azure/container-linux/kubernetes/workers/versions.tf
+++ b/azure/container-linux/kubernetes/workers/versions.tf
@ -0,0 +1,4 @@
+
+terraform {
+  required_version = ">= 0.12"
+}
--- a/azure/container-linux/kubernetes/workers/workers.tf
+++ b/azure/container-linux/kubernetes/workers/workers.tf
@ -1,28 +1,28 @@
 locals {
  # Channel for a Container Linux derivative
  # coreos-stable -> Container Linux Stable
-  channel = "${element(split("-", var.os_image), 1)}"
+  channel = element(split("-", var.os_image), 1)
 }

 # Workers scale set
 resource "azurerm_virtual_machine_scale_set" "workers" {
-  resource_group_name = "${var.resource_group_name}"
+  resource_group_name = var.resource_group_name

  name                   = "${var.name}-workers"
-  location               = "${var.region}"
+  location               = var.region
  single_placement_group = false

  sku {
-    name     = "${var.vm_type}"
+    name     = var.vm_type
    tier     = "standard"
-    capacity = "${var.count}"
+    capacity = var.worker_count
  }

  # boot
  storage_profile_image_reference {
    publisher = "CoreOS"
    offer     = "CoreOS"
-    sku       = "${local.channel}"
+    sku       = local.channel
    version   = "latest"
  }

@ -37,7 +37,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
  os_profile {
    computer_name_prefix = "${var.name}-worker-"
    admin_username       = "core"
-    custom_data          = "${data.ct_config.worker-ignition.rendered}"
+    custom_data          = data.ct_config.worker-ignition.rendered
  }

  # Azure mandates setting an ssh_key, even though Ignition custom_data handles it too
@ -46,7 +46,7 @@ resource "azurerm_virtual_machine_scale_set" "workers" {

    ssh_keys {
      path     = "/home/core/.ssh/authorized_keys"
-      key_data = "${var.ssh_authorized_key}"
+      key_data = var.ssh_authorized_key
    }
  }

@ -54,61 +54,63 @@ resource "azurerm_virtual_machine_scale_set" "workers" {
  network_profile {
    name                      = "nic0"
    primary                   = true
-    network_security_group_id = "${var.security_group_id}"
+    network_security_group_id = var.security_group_id

    ip_configuration {
      name      = "ip0"
      primary   = true
-      subnet_id = "${var.subnet_id}"
+      subnet_id = var.subnet_id

      # backend address pool to which the NIC should be added
-      load_balancer_backend_address_pool_ids = ["${var.backend_address_pool_id}"]
+      load_balancer_backend_address_pool_ids = [var.backend_address_pool_id]
    }
  }

  # lifecycle
  upgrade_policy_mode = "Manual"
-  priority            = "${var.priority}"
-  eviction_policy     = "Delete"
+  # eviction policy may only be set when priority is Low
+  priority            = var.priority
+  eviction_policy     = var.priority == "Low" ? "Delete" : null
 }

 # Scale up or down to maintain desired number, tolerating deallocations.
-resource "azurerm_autoscale_setting" "workers" {
-  resource_group_name = "${var.resource_group_name}"
+resource "azurerm_monitor_autoscale_setting" "workers" {
+  resource_group_name = var.resource_group_name

  name     = "${var.name}-maintain-desired"
-  location = "${var.region}"
+  location = var.region

  # autoscale
  enabled            = true
-  target_resource_id = "${azurerm_virtual_machine_scale_set.workers.id}"
+  target_resource_id = azurerm_virtual_machine_scale_set.workers.id

  profile {
    name = "default"

    capacity {
-      minimum = "${var.count}"
-      default = "${var.count}"
-      maximum = "${var.count}"
+      minimum = var.worker_count
+      default = var.worker_count
+      maximum = var.worker_count
    }
  }
 }

 # Worker Ignition configs
 data "ct_config" "worker-ignition" {
-  content      = "${data.template_file.worker-config.rendered}"
+  content      = data.template_file.worker-config.rendered
  pretty_print = false
-  snippets     = ["${var.clc_snippets}"]
+  snippets     = var.clc_snippets
 }

 # Worker Container Linux configs
 data "template_file" "worker-config" {
-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+  template = file("${path.module}/cl/worker.yaml.tmpl")

  vars = {
-    kubeconfig             = "${indent(10, var.kubeconfig)}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
-    cluster_dns_service_ip = "${cidrhost(var.service_cidr, 10)}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
+    kubeconfig             = indent(10, var.kubeconfig)
+    ssh_authorized_key     = var.ssh_authorized_key
+    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
+    cluster_domain_suffix  = var.cluster_domain_suffix
  }
 }
+
--- a/bare-metal/container-linux/kubernetes/README.md
+++ b/bare-metal/container-linux/kubernetes/README.md
@ -11,7 +11,7 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.13.4 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
+* Kubernetes v1.15.0 (upstream, via [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube))
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
 * Advanced features like [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
--- a/bare-metal/container-linux/kubernetes/bootkube.tf
+++ b/bare-metal/container-linux/kubernetes/bootkube.tf
@ -1,16 +1,18 @@
 # Self-hosted Kubernetes assets (kubeconfig, manifests)
 module "bootkube" {
-  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=953521dbba49eb6a39204f30a3978730eac01e11"
+  source = "git::https://github.com/poseidon/terraform-render-bootkube.git?ref=62df9ad69cc0da35f47d40fa981370c4503ad581"

-  cluster_name                    = "${var.cluster_name}"
-  api_servers                     = ["${var.k8s_domain_name}"]
-  etcd_servers                    = ["${var.controller_domains}"]
-  asset_dir                       = "${var.asset_dir}"
-  networking                      = "${var.networking}"
-  network_mtu                     = "${var.network_mtu}"
-  network_ip_autodetection_method = "${var.network_ip_autodetection_method}"
-  pod_cidr                        = "${var.pod_cidr}"
-  service_cidr                    = "${var.service_cidr}"
-  cluster_domain_suffix           = "${var.cluster_domain_suffix}"
-  enable_reporting                = "${var.enable_reporting}"
+  cluster_name                    = var.cluster_name
+  api_servers                     = [var.k8s_domain_name]
+  etcd_servers                    = var.controller_domains
+  asset_dir                       = var.asset_dir
+  networking                      = var.networking
+  network_mtu                     = var.network_mtu
+  network_ip_autodetection_method = var.network_ip_autodetection_method
+  pod_cidr                        = var.pod_cidr
+  service_cidr                    = var.service_cidr
+  cluster_domain_suffix           = var.cluster_domain_suffix
+  enable_reporting                = var.enable_reporting
+  enable_aggregation              = var.enable_aggregation
 }
+
--- a/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/controller.yaml.tmpl
@ -7,7 +7,7 @@ systemd:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.3.12"
+            Environment="ETCD_IMAGE_TAG=v3.3.13"
            Environment="ETCD_NAME=${etcd_name}"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${domain_name}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${domain_name}:2380"
@ -75,6 +75,7 @@ systemd:
          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=iscsiadm,target=/sbin/iscsiadm \
          --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@ -89,6 +90,7 @@ systemd:
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -128,7 +130,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.4
+          KUBELET_IMAGE_TAG=v1.15.0
    - path: /etc/hostname
      filesystem: root
      mode: 0644
@ -160,7 +162,7 @@ storage:
            --mount volume=assets,target=/assets \
            --volume bootstrap,kind=host,source=/etc/kubernetes \
            --mount volume=bootstrap,target=/etc/kubernetes \
-            $$RKT_OPTS \
+            $${RKT_OPTS} \
            quay.io/coreos/bootkube:v0.14.0 \
            --net=host \
            --dns=host \
--- a/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
+++ b/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl
@ -50,6 +50,7 @@ systemd:
          --volume iscsiadm,kind=host,source=/usr/sbin/iscsiadm \
          --mount volume=iscsiadm,target=/sbin/iscsiadm \
          --insecure-options=image"
+        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
@ -62,6 +63,7 @@ systemd:
          --anonymous-auth=false \
          --authentication-token-webhook \
          --authorization-mode=Webhook \
+          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${cluster_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
@ -89,7 +91,7 @@ storage:
      contents:
        inline: |
          KUBELET_IMAGE_URL=docker://k8s.gcr.io/hyperkube
-          KUBELET_IMAGE_TAG=v1.13.4
+          KUBELET_IMAGE_TAG=v1.15.0
    - path: /etc/hostname
      filesystem: root
      mode: 0644
--- a/bare-metal/container-linux/kubernetes/groups.tf
+++ b/bare-metal/container-linux/kubernetes/groups.tf
@ -1,33 +1,35 @@
 resource "matchbox_group" "install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)

-  name = "${format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))}"
+  name = format("install-%s", element(concat(var.controller_names, var.worker_names), count.index))

-  profile = "${local.flavor == "flatcar" ? var.cached_install == "true" ? element(matchbox_profile.cached-flatcar-linux-install.*.name, count.index) : element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)}"
+  # pick one of 4 Matchbox profiles (Container Linux or Flatcar, cached or non-cached)
+  profile = local.flavor == "flatcar" ? var.cached_install == "true" ? element(matchbox_profile.cached-flatcar-linux-install.*.name, count.index) : element(matchbox_profile.flatcar-install.*.name, count.index) : var.cached_install == "true" ? element(matchbox_profile.cached-container-linux-install.*.name, count.index) : element(matchbox_profile.container-linux-install.*.name, count.index)

-  selector {
-    mac = "${element(concat(var.controller_macs, var.worker_macs), count.index)}"
+  selector = {
+    mac = element(concat(var.controller_macs, var.worker_macs), count.index)
  }
 }

 resource "matchbox_group" "controller" {
-  count   = "${length(var.controller_names)}"
-  name    = "${format("%s-%s", var.cluster_name, element(var.controller_names, count.index))}"
-  profile = "${element(matchbox_profile.controllers.*.name, count.index)}"
+  count   = length(var.controller_names)
+  name    = format("%s-%s", var.cluster_name, element(var.controller_names, count.index))
+  profile = element(matchbox_profile.controllers.*.name, count.index)

-  selector {
-    mac = "${element(var.controller_macs, count.index)}"
+  selector = {
+    mac = element(var.controller_macs, count.index)
    os  = "installed"
  }
 }

 resource "matchbox_group" "worker" {
-  count   = "${length(var.worker_names)}"
-  name    = "${format("%s-%s", var.cluster_name, element(var.worker_names, count.index))}"
-  profile = "${element(matchbox_profile.workers.*.name, count.index)}"
+  count   = length(var.worker_names)
+  name    = format("%s-%s", var.cluster_name, element(var.worker_names, count.index))
+  profile = element(matchbox_profile.workers.*.name, count.index)

-  selector {
-    mac = "${element(var.worker_macs, count.index)}"
+  selector = {
+    mac = element(var.worker_macs, count.index)
    os  = "installed"
  }
 }
+
--- a/bare-metal/container-linux/kubernetes/outputs.tf
+++ b/bare-metal/container-linux/kubernetes/outputs.tf
@ -1,3 +1,4 @@
 output "kubeconfig-admin" {
-  value = "${module.bootkube.kubeconfig-admin}"
+  value = module.bootkube.kubeconfig-admin
 }
+
--- a/bare-metal/container-linux/kubernetes/profiles.tf
+++ b/bare-metal/container-linux/kubernetes/profiles.tf
@ -1,48 +1,47 @@
 locals {
  # coreos-stable -> coreos flavor, stable channel
  # flatcar-stable -> flatcar flavor, stable channel
-  flavor = "${element(split("-", var.os_channel), 0)}"
+  flavor = element(split("-", var.os_channel), 0)

-  channel = "${element(split("-", var.os_channel), 1)}"
+  channel = element(split("-", var.os_channel), 1)
 }

 // Container Linux Install profile (from release.core-os.net)
 resource "matchbox_profile" "container-linux-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-  name  = "${format("%s-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  count = length(var.controller_names) + length(var.worker_names)
+  name  = format("%s-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))

-  kernel = "http://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe.vmlinuz"
+  kernel = "${var.download_protocol}://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe.vmlinuz"

  initrd = [
-    "http://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe_image.cpio.gz",
+    "${var.download_protocol}://${local.channel}.release.core-os.net/amd64-usr/${var.os_version}/coreos_production_pxe_image.cpio.gz",
  ]

-  args = [
+  args = flatten([
    "initrd=coreos_production_pxe_image.cpio.gz",
    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.first_boot=yes",
    "console=tty0",
    "console=ttyS0",
-    "${var.kernel_args}",
-  ]
+    var.kernel_args,
+  ])

-  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.container-linux-install-configs.*.rendered, count.index)
 }

 data "template_file" "container-linux-install-configs" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)

-  template = "${file("${path.module}/cl/install.yaml.tmpl")}"
-
-  vars {
-    os_flavor           = "${local.flavor}"
-    os_channel          = "${local.channel}"
-    os_version          = "${var.os_version}"
-    ignition_endpoint   = "${format("%s/ignition", var.matchbox_http_endpoint)}"
-    install_disk        = "${var.install_disk}"
-    container_linux_oem = "${var.container_linux_oem}"
-    ssh_authorized_key  = "${var.ssh_authorized_key}"
+  template = file("${path.module}/cl/install.yaml.tmpl")

+  vars = {
+    os_flavor           = local.flavor
+    os_channel          = local.channel
+    os_version          = var.os_version
+    ignition_endpoint   = format("%s/ignition", var.matchbox_http_endpoint)
+    install_disk        = var.install_disk
+    container_linux_oem = var.container_linux_oem
+    ssh_authorized_key  = var.ssh_authorized_key
    # only cached-container-linux profile adds -b baseurl
    baseurl_flag = ""
  }
@ -51,8 +50,8 @@ data "template_file" "container-linux-install-configs" {
 // Container Linux Install profile (from matchbox /assets cache)
 // Note: Admin must have downloaded os_version into matchbox assets/coreos.
 resource "matchbox_profile" "cached-container-linux-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-  name  = "${format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  count = length(var.controller_names) + length(var.worker_names)
+  name  = format("%s-cached-container-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))

  kernel = "/assets/coreos/${var.os_version}/coreos_production_pxe.vmlinuz"

@ -60,32 +59,31 @@ resource "matchbox_profile" "cached-container-linux-install" {
    "/assets/coreos/${var.os_version}/coreos_production_pxe_image.cpio.gz",
  ]

-  args = [
+  args = flatten([
    "initrd=coreos_production_pxe_image.cpio.gz",
    "coreos.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "coreos.first_boot=yes",
    "console=tty0",
    "console=ttyS0",
-    "${var.kernel_args}",
-  ]
+    var.kernel_args,
+  ])

-  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)
 }

 data "template_file" "cached-container-linux-install-configs" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
+  count = length(var.controller_names) + length(var.worker_names)

-  template = "${file("${path.module}/cl/install.yaml.tmpl")}"
-
-  vars {
-    os_flavor           = "${local.flavor}"
-    os_channel          = "${local.channel}"
-    os_version          = "${var.os_version}"
-    ignition_endpoint   = "${format("%s/ignition", var.matchbox_http_endpoint)}"
-    install_disk        = "${var.install_disk}"
-    container_linux_oem = "${var.container_linux_oem}"
-    ssh_authorized_key  = "${var.ssh_authorized_key}"
+  template = file("${path.module}/cl/install.yaml.tmpl")

+  vars = {
+    os_flavor           = local.flavor
+    os_channel          = local.channel
+    os_version          = var.os_version
+    ignition_endpoint   = format("%s/ignition", var.matchbox_http_endpoint)
+    install_disk        = var.install_disk
+    container_linux_oem = var.container_linux_oem
+    ssh_authorized_key  = var.ssh_authorized_key
    # profile uses -b baseurl to install from matchbox cache
    baseurl_flag = "-b ${var.matchbox_http_endpoint}/assets/${local.flavor}"
  }
@ -93,32 +91,32 @@ data "template_file" "cached-container-linux-install-configs" {

 // Flatcar Linux install profile (from release.flatcar-linux.net)
 resource "matchbox_profile" "flatcar-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-  name  = "${format("%s-flatcar-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  count = length(var.controller_names) + length(var.worker_names)
+  name  = format("%s-flatcar-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))

-  kernel = "http://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe.vmlinuz"
+  kernel = "${var.download_protocol}://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe.vmlinuz"

  initrd = [
-    "http://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
+    "${var.download_protocol}://${local.channel}.release.flatcar-linux.net/amd64-usr/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
  ]

-  args = [
+  args = flatten([
    "initrd=flatcar_production_pxe_image.cpio.gz",
    "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "flatcar.first_boot=yes",
    "console=tty0",
    "console=ttyS0",
-    "${var.kernel_args}",
-  ]
+    var.kernel_args,
+  ])

-  container_linux_config = "${element(data.template_file.container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.container-linux-install-configs.*.rendered, count.index)
 }

 // Flatcar Linux Install profile (from matchbox /assets cache)
 // Note: Admin must have downloaded os_version into matchbox assets/flatcar.
 resource "matchbox_profile" "cached-flatcar-linux-install" {
-  count = "${length(var.controller_names) + length(var.worker_names)}"
-  name  = "${format("%s-cached-flatcar-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))}"
+  count = length(var.controller_names) + length(var.worker_names)
+  name  = format("%s-cached-flatcar-linux-install-%s", var.cluster_name, element(concat(var.controller_names, var.worker_names), count.index))

  kernel = "/assets/flatcar/${var.os_version}/flatcar_production_pxe.vmlinuz"

@ -126,90 +124,93 @@ resource "matchbox_profile" "cached-flatcar-linux-install" {
    "/assets/flatcar/${var.os_version}/flatcar_production_pxe_image.cpio.gz",
  ]

-  args = [
+  args = flatten([
    "initrd=flatcar_production_pxe_image.cpio.gz",
    "flatcar.config.url=${var.matchbox_http_endpoint}/ignition?uuid=$${uuid}&mac=$${mac:hexhyp}",
    "flatcar.first_boot=yes",
    "console=tty0",
    "console=ttyS0",
-    "${var.kernel_args}",
-  ]
+    var.kernel_args,
+  ])

-  container_linux_config = "${element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)}"
+  container_linux_config = element(data.template_file.cached-container-linux-install-configs.*.rendered, count.index)
 }

 // Kubernetes Controller profiles
 resource "matchbox_profile" "controllers" {
-  count        = "${length(var.controller_names)}"
-  name         = "${format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))}"
-  raw_ignition = "${element(data.ct_config.controller-ignitions.*.rendered, count.index)}"
+  count        = length(var.controller_names)
+  name         = format("%s-controller-%s", var.cluster_name, element(var.controller_names, count.index))
+  raw_ignition = element(data.ct_config.controller-ignitions.*.rendered, count.index)
 }

 data "ct_config" "controller-ignitions" {
-  count        = "${length(var.controller_names)}"
-  content      = "${element(data.template_file.controller-configs.*.rendered, count.index)}"
+  count        = length(var.controller_names)
+  content      = element(data.template_file.controller-configs.*.rendered, count.index)
  pretty_print = false
-
-  # Must use direct lookup. Cannot use lookup(map, key) since it only works for flat maps
-  snippets = ["${local.clc_map[element(var.controller_names, count.index)]}"]
+  snippets     = local.clc_map[element(var.controller_names, count.index)]
 }

 data "template_file" "controller-configs" {
-  count = "${length(var.controller_names)}"
+  count = length(var.controller_names)

-  template = "${file("${path.module}/cl/controller.yaml.tmpl")}"
+  template = file("${path.module}/cl/controller.yaml.tmpl")

-  vars {
-    domain_name            = "${element(var.controller_domains, count.index)}"
-    etcd_name              = "${element(var.controller_names, count.index)}"
-    etcd_initial_cluster   = "${join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))}"
-    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+  vars = {
+    domain_name            = element(var.controller_domains, count.index)
+    etcd_name              = element(var.controller_names, count.index)
+    etcd_initial_cluster   = join(",", formatlist("%s=https://%s:2380", var.controller_names, var.controller_domains))
+    cgroup_driver = var.os_channel == "flatcar-edge" ? "systemd" : "cgroupfs"
+    cluster_dns_service_ip = module.bootkube.cluster_dns_service_ip
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    ssh_authorized_key     = var.ssh_authorized_key
  }
 }

 // Kubernetes Worker profiles
 resource "matchbox_profile" "workers" {
-  count        = "${length(var.worker_names)}"
-  name         = "${format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))}"
-  raw_ignition = "${element(data.ct_config.worker-ignitions.*.rendered, count.index)}"
+  count        = length(var.worker_names)
+  name         = format("%s-worker-%s", var.cluster_name, element(var.worker_names, count.index))
+  raw_ignition = element(data.ct_config.worker-ignitions.*.rendered, count.index)
 }

 data "ct_config" "worker-ignitions" {
-  count        = "${length(var.worker_names)}"
-  content      = "${element(data.template_file.worker-configs.*.rendered, count.index)}"
+  count        = length(var.worker_names)
+  content      = element(data.template_file.worker-configs.*.rendered, count.index)
  pretty_print = false
-
-  # Must use direct lookup. Cannot use lookup(map, key) since it only works for flat maps
-  snippets = ["${local.clc_map[element(var.worker_names, count.index)]}"]
+  snippets     = local.clc_map[element(var.worker_names, count.index)]
 }

 data "template_file" "worker-configs" {
-  count = "${length(var.worker_names)}"
+  count = length(var.worker_names)

-  template = "${file("${path.module}/cl/worker.yaml.tmpl")}"
+  template = file("${path.module}/cl/worker.yaml.tmpl")

-  vars {
-    domain_name            = "${element(var.worker_domains, count.index)}"
-    cluster_dns_service_ip = "${module.bootkube.cluster_dns_service_ip}"
-    cluster_domain_suffix  = "${var.cluster_domain_suffix}"
-    ssh_authorized_key     = "${var.ssh_authorized_key}"
+  vars = {
+    domain_name            = element(var.worker_domains, count.index)
+    cgroup_driver = var.os_channel == "flatcar-edge" ? "systemd" : "cgroupfs"
+    cluster_dns_service_ip = module.bootkube.cluster_dns_service_ip
+    cluster_domain_suffix  = var.cluster_domain_suffix
+    ssh_authorized_key     = var.ssh_authorized_key
  }
 }

 locals {
  # Hack to workaround https://github.com/hashicorp/terraform/issues/17251
+  # Still an issue in Terraform v0.12 https://github.com/hashicorp/terraform/issues/20572
  # Default Container Linux config snippets map every node names to list("\n") so
  # all lookups succeed
-  clc_defaults = "${zipmap(concat(var.controller_names, var.worker_names), chunklist(data.template_file.clc-default-snippets.*.rendered, 1))}"
+  clc_defaults = zipmap(
+    concat(var.controller_names, var.worker_names),
+    chunklist(data.template_file.clc-default-snippets.*.rendered, 1),
+  )

  # Union of the default and user specific snippets, later overrides prior.
-  clc_map = "${merge(local.clc_defaults, var.clc_snippets)}"
+  clc_map = merge(local.clc_defaults, var.clc_snippets)
 }

 // Horrible hack to generate a Terraform list of node count length
 data "template_file" "clc-default-snippets" {
-  count    = "${length(var.controller_names) + length(var.worker_names)}"
+  count    = length(var.controller_names) + length(var.worker_names)
  template = "\n"
 }
+
--- a/bare-metal/container-linux/kubernetes/require.tf
+++ b/bare-metal/container-linux/kubernetes/require.tf
@ -1,21 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = ">= 0.11.0"
-}
-
-provider "local" {
-  version = "~> 1.0"
-}
-
-provider "null" {
-  version = "~> 1.0"
-}
-
-provider "template" {
-  version = "~> 1.0"
-}
-
-provider "tls" {
-  version = "~> 1.0"
-}
--- a/bare-metal/container-linux/kubernetes/ssh.tf
+++ b/bare-metal/container-linux/kubernetes/ssh.tf
@ -1,59 +1,59 @@
 # Secure copy etcd TLS assets and kubeconfig to controllers. Activates kubelet.service
 resource "null_resource" "copy-controller-secrets" {
-  count = "${length(var.controller_names)}"
+  count = length(var.controller_names)

  # Without depends_on, remote-exec could start and wait for machines before
  # matchbox groups are written, causing a deadlock.
  depends_on = [
-    "matchbox_group.install",
-    "matchbox_group.controller",
-    "matchbox_group.worker",
+    matchbox_group.install,
+    matchbox_group.controller,
+    matchbox_group.worker,
  ]

  connection {
    type    = "ssh"
-    host    = "${element(var.controller_domains, count.index)}"
+    host    = element(var.controller_domains, count.index)
    user    = "core"
    timeout = "60m"
  }

  provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
+    content     = module.bootkube.kubeconfig-kubelet
    destination = "$HOME/kubeconfig"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_ca_cert}"
+    content     = module.bootkube.etcd_ca_cert
    destination = "$HOME/etcd-client-ca.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_cert}"
+    content     = module.bootkube.etcd_client_cert
    destination = "$HOME/etcd-client.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_client_key}"
+    content     = module.bootkube.etcd_client_key
    destination = "$HOME/etcd-client.key"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_cert}"
+    content     = module.bootkube.etcd_server_cert
    destination = "$HOME/etcd-server.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_server_key}"
+    content     = module.bootkube.etcd_server_key
    destination = "$HOME/etcd-server.key"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_cert}"
+    content     = module.bootkube.etcd_peer_cert
    destination = "$HOME/etcd-peer.crt"
  }

  provisioner "file" {
-    content     = "${module.bootkube.etcd_peer_key}"
+    content     = module.bootkube.etcd_peer_key
    destination = "$HOME/etcd-peer.key"
  }

@ -76,25 +76,25 @@ resource "null_resource" "copy-controller-secrets" {

 # Secure copy kubeconfig to all workers. Activates kubelet.service
 resource "null_resource" "copy-worker-secrets" {
-  count = "${length(var.worker_names)}"
+  count = length(var.worker_names)

  # Without depends_on, remote-exec could start and wait for machines before
  # matchbox groups are written, causing a deadlock.
  depends_on = [
-    "matchbox_group.install",
-    "matchbox_group.controller",
-    "matchbox_group.worker",
+    matchbox_group.install,
+    matchbox_group.controller,
+    matchbox_group.worker,
  ]

  connection {
    type    = "ssh"
-    host    = "${element(var.worker_domains, count.index)}"
+    host    = element(var.worker_domains, count.index)
    user    = "core"
    timeout = "60m"
  }

  provisioner "file" {
-    content     = "${module.bootkube.kubeconfig-kubelet}"
+    content     = module.bootkube.kubeconfig-kubelet
    destination = "$HOME/kubeconfig"
  }

@ -112,19 +112,19 @@ resource "null_resource" "bootkube-start" {
  # Terraform only does one task at a time, so it would try to bootstrap
  # while no Kubelets are running.
  depends_on = [
-    "null_resource.copy-controller-secrets",
-    "null_resource.copy-worker-secrets",
+    null_resource.copy-controller-secrets,
+    null_resource.copy-worker-secrets,
  ]

  connection {
    type    = "ssh"
-    host    = "${element(var.controller_domains, 0)}"
+    host    = element(var.controller_domains, 0)
    user    = "core"
    timeout = "15m"
  }

  provisioner "file" {
-    source      = "${var.asset_dir}"
+    source      = var.asset_dir
    destination = "$HOME/assets"
  }

@ -135,3 +135,4 @@ resource "null_resource" "bootkube-start" {
    ]
  }
 }
+
--- a/Show More
+++ b/Show More