Update Calico from v3.24.5 to v3.25.0

* https://github.com/poseidon/terraform-render-bootstrap/pull/342
Update etcd from v3.5.6 to v3.5.7
2025-08-02 15:31:35 +02:00 · 2023-01-24 08:29:08 -08:00 · 2023-01-24 08:29:08 -08:00 · 2023-01-23 09:08:20 -08:00 · 2023-01-23 09:02:30 -08:00 · 2023-01-19 08:54:46 -08:00
239 changed files with 21443 additions and 8765 deletions
--- a/.github/FUNDING.yml
+++ b/.github/FUNDING.yml
@ -0,0 +1 @@
+github: [poseidon]
--- a/.github/dependabot.yaml
+++ b/.github/dependabot.yaml
@ -0,0 +1,9 @@
+version: 2
+updates:
+- package-ecosystem: pip
+  directory: "/"
+  schedule:
+    interval: weekly
+  pull-request-branch-name:
+    separator: "-"
+  open-pull-requests-limit: 3
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1 @@
+site/
--- a/CHANGES.md
+++ b/CHANGES.md
@ -4,6 +4,882 @@ Notable changes between versions.

 ## Latest

+## v1.26.1
+
+* Kubernetes [v1.26.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#v1261)
+* Update etcd from v3.5.6 to [v3.5.7](https://github.com/etcd-io/etcd/releases/tag/v3.5.7)
+* Update Cilium from v1.12.4 to [v1.12.5](https://github.com/cilium/cilium/releases/tag/v1.12.5)
+* Update Calico from v3.24.5 to [v3.25.0](https://github.com/projectcalico/calico/releases/tag/v3.25.0)
+* Update CoreDNS from v1.9.3 to [v1.9.4](https://github.com/poseidon/terraform-render-bootstrap/pull/341)
+
+## v1.26.0
+
+* Kubernetes [v1.26.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#v1260)
+* Update etcd from v3.5.5 to [v3.5.6](https://github.com/etcd-io/etcd/releases/tag/v3.5.6)
+* Update Cilium from v1.12.3 to [v1.12.4](https://github.com/cilium/cilium/releases/tag/v1.12.4)
+* Update flannel from v0.15.1 to [v0.20.2](https://github.com/flannel-io/flannel/releases/tag/v0.20.2)
+* Reminder: Modules are no longer published to the [Terraform Module Registry](https://registry.terraform.io/search/modules?q=poseidon) ([#1282](https://github.com/poseidon/typhoon/pull/1282))
+  * See [#1282](https://github.com/poseidon/typhoon/pull/1282) and [v1.25.4](https://github.com/poseidon/typhoon/releases/tag/v1.25.4) for details
+
+### AWS
+
+* Migrate AWS launch configurations to launch templates ([#1275](https://github.com/poseidon/typhoon/pull/1275))
+  * Starting Dec 31, 2022 AWS won't add new instance types/families to launch configurations
+
+### Addons
+
+* Update ingress-nginx from v1.3.1 to [v1.5.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.5.1)
+* Update Prometheus from v2.40.1 to [v2.40.5](https://github.com/prometheus/prometheus/releases/tag/v2.40.5)
+* Update node-exporter from v1.3.1 to [v1.5.0](https://github.com/prometheus/node_exporter/releases/tag/v1.5.0)
+* Update kube-state-metrics from v2.6.0 to [v2.7.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.7.0)
+* Update Grafana from v9.2.4 to [v9.3.1](https://github.com/grafana/grafana/releases/tag/v9.3.1)
+
+## v1.25.4
+
+* Kubernetes [v1.25.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1254)
+* Update Calico from v3.24.1 to [v3.24.5](https://github.com/projectcalico/calico/releases/tag/v3.24.5)
+* Allow Kubelet kubeconfig to drain nodes, if desired ([#330](https://github.com/poseidon/terraform-render-bootstrap/pull/330))
+* Re-enable Kubelet Graceful Node Shutdown ([#1261](https://github.com/poseidon/typhoon/pull/1261))
+  * Introduce companion project [poseidon/scuttle](https://github.com/poseidon/scuttle)
+* Link to new Mastodon account for release announcements
+  * [@typhoon@fosstodon.org](https://fosstodon.org/@typhoon)
+  * [@poseidon@fosstodon.org](https://fosstodon.org/@poseidon)
+* Deprecate publishing to the [Terraform Module Registry](https://registry.terraform.io/search/modules?q=poseidon)
+  * Typhoon docs have always shown using Git-based module sources, not the Terraform Module Registry
+  * Module usage should be `source = "git::https://github.com/poseidon/typhoon/...` not `source = poseidon/kubernetes/...`
+  * Terraform's Module Registry requires subtree mirroring typhoon to special terraform-platform-kubernetes repos, only supports release versions (no commit SHAs or forks), only ever contained Flatcar Linux modules (not Fedora CoreOS) for historical reasons
+  * Note, this does not affect Terraform Providers like `poseidon/matchbox` or `poseidon/ct`, the registry works well for providers
+
+### Fedora CoreOS
+
+* Remove unused `Wants=network.target` from `etcd-member.service` ([#1254](https://github.com/poseidon/typhoon/pull/1254))
+
+### Cloud
+
+* Remove defunct `delete-node.service` from worker node configurations ([#1256](https://github.com/poseidon/typhoon/pull/1256))
+
+### Addons
+
+* Update Prometheus from v2.39.1 to [v2.40.1](https://github.com/prometheus/prometheus/releases/tag/v2.40.1)
+* Update Grafana from v9.1.7 to [v9.2.4](https://github.com/grafana/grafana/releases/tag/v9.2.4)
+
+## v1.25.3
+
+* Kubernetes [v1.25.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1253)
+* Switch Kubernetes registry from `k8s.gcr.io` to `registry.k8s.io` for addons ([#1246](https://github.com/poseidon/typhoon/pull/1246))
+* Update Cilium from v1.12.2 to [v1.12.3](https://github.com/cilium/cilium/releases/tag/v1.12.3) ([#1253](https://github.com/poseidon/typhoon/pull/1253))
+
+### Azure
+
+* Change default Azure `worker_type` from [`Standard_DS1_v2`](https://learn.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series#dsv2-series) to [`Standard_D2as_v5`](https://learn.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series#dasv5-series) ([#1248](https://github.com/poseidon/typhoon/pull/1248))
+  * Get 2 VCPU, 7 GiB, 12500Mbps (vs 1 VCPU, 3.5GiB, 750 Mbps)
+  * Small increase in pay-as-you-go price ($53.29 -> $62.78)
+  * Small increase in spot price ($5.64/mo -> $7.37/mo)
+  * Change from Intel to AMD EPYC (`D2as_v5` cheaper than `D2s_v5`)
+
+### Flatcar Linux
+
+* Add Flatcar Linux ARM64 support on Azure ([docs](https://typhoon.psdn.io/advanced/arm64/), [#1251](https://github.com/poseidon/typhoon/pull/1251))
+* Switch from Azure Hypervisor gen1 to gen2 (**action required**) ([#1248](https://github.com/poseidon/typhoon/pull/1248))
+  * Run `az vm image terms accept --publish kinvolk --offer flatcar-container-linux-free --plan stable-gen2`
+
+### Docs
+
+* Remove old docs note about not supporting ARM64 with Calico
+  * Typhoon supports ARM64 with `cilium`, `calico`, and `flannel`
+
+### Addons
+
+* Update Prometheus from v2.38.0 to [v2.39.1](https://github.com/prometheus/prometheus/releases/tag/v2.39.1)
+* Update Grafana from v9.1.6 to [v9.1.7](https://github.com/grafana/grafana/releases/tag/v9.1.7)
+
+## v1.25.2
+
+Kubernetes v1.25.2 was skipped since there were minimal changes upstream.
+
+## v1.25.1
+
+* Kubernetes [v1.25.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1251)
+* Update etcd from v3.5.4 to [v3.5.5](https://github.com/etcd-io/etcd/releases/tag/v3.5.5)
+* Update Cilium from v1.12.1 to [v1.12.2](https://github.com/cilium/cilium/releases/tag/v1.12.2)
+* Update Calico from v3.23.3 to [v3.24.1](https://github.com/projectcalico/calico/releases/tag/v3.24.1)
+* Revert Kubelet Graceful Node Shutdown on worker nodes ([#1227](https://github.com/poseidon/typhoon/pull/1227))
+  * Fix issue where non-critical pods are left in Error/Completed state on node shutdown
+* Remove feature flag disable workaround for [kubernetes/kubernetes#112081](https://github.com/kubernetes/kubernetes/issues/112081)
+  * Kubernetes [reverted](https://github.com/kubernetes/kubernetes/pull/112078) `LocalStorageCapacityIsolationFSQuotaMonitoring` back to alpha
+* Remove workaround for preventing `search .` propagation in [kubernetes/kubernetes#112135](https://github.com/kubernetes/kubernetes/issues/112135)
+  * Upstream Kubernetes [fix](https://github.com/kubernetes/kubernetes/pull/112157)
+
+### Addons
+
+* Update kube-state-metrics from v2.5.0 to [v2.6.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.6.0)
+* Update ingress-nginx from v1.3.0 to [v1.3.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.3.1)
+* Update Grafana from v9.1.0 to [v9.1.6](https://github.com/grafana/grafana/releases/tag/v9.1.6)
+
+## v1.25.0
+
+* Kubernetes [v1.25.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#v1250)
+  * Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature gate ([#1220](https://github.com/poseidon/typhoon/pull/1220), fixes [kubernetes#112081](https://github.com/kubernetes/kubernetes/issues/112081))
+  * Add workaround to revert adding "search ." to containers' `/etc/resolv.conf` ([#1224](https://github.com/poseidon/typhoon/pull/1224), fixes [kubernetes#112135](https://github.com/kubernetes/kubernetes/issues/112135))
+* Migrate most Kubelet flags to KubeletConfiguration file ([#1219](https://github.com/poseidon/typhoon/pull/1219))
+* Configure Kubelet Graceful Node Shutdown ([#1222](https://github.com/poseidon/typhoon/pull/1222))
+  * Allow up to 30s for critical pods to gracefully shutdown on node shutdown
+  * Allow up to 15s for regular pods to gracefully shutdown on node shutdown
+  * Mark node NotReady promptly on node shutdown
+  * Lengthen systemd inhibitor lock max delay from 5s to 45s
+
+### Fedora CoreOS
+
+* Change Podman `log-driver` from `journald` to `k8s-file` ([#1221](https://github.com/poseidon/typhoon/pull/1221))
+  * Fix `etcd-member` and Kubelet systemd service log lines appearing twice in journal logs
+
+## v1.24.4
+
+* Kubernetes [v1.24.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1244)
+* Update CoreDNS from v1.8.6 to [v1.9.3](https://github.com/poseidon/terraform-render-bootstrap/pull/318)
+* Update Cilium from v1.11.7 to [v1.12.1](https://github.com/cilium/cilium/releases/tag/v1.12.1)
+* Update Calico from v3.23.1 to [v3.23.3](https://github.com/projectcalico/calico/releases/tag/v3.23.3)
+* Switch Kubernetes registry from `k8s.gcr.io` to `registry.k8s.io` ([#1206](https://github.com/poseidon/typhoon/pull/1206))
+* Remove use of deprecated Terraform [template](https://registry.terraform.io/providers/hashicorp/template) provider ([#1194](https://github.com/poseidon/typhoon/pull/1194))
+
+### Fedora CoreOS
+
+* Remove ineffective `/etc/fedora-coreos/iptables-legacy.stamp` ([#1201](https://github.com/poseidon/typhoon/pull/1201))
+  * Typhoon already uses iptables v1.8.7 (nf_tables) since FCOS 36
+  * Staying on legacy iptables required a file in `/etc/coreos` instead
+
+### Flatcar Linux
+
+* Migrate Flatcar Linux from Ignition spec v2.3.0 to v3.3.0 ([#1196](https://github.com/poseidon/typhoon/pull/1196)) (**action required**)
+  * Flatcar Linux 3185.0.0+ [supports](https://flatcar-linux.org/docs/latest/provisioning/ignition/specification/#ignition-v3) Ignition v3.x specs (which are rendered from Butane Configs, like Fedora CoreOS)
+  * `poseidon/ct` v0.11.0 [supports](https://github.com/poseidon/terraform-provider-ct/pull/131) the `flatcar` Butane Config variant
+  * Require poseidon/ct v0.11+ and Flatcar Linux 3185.0.0+
+* Please modify any Flatcar Linux snippets to use the [Butane Config](https://coreos.github.io/butane/config-flatcar-v1_0/) format (**action required**)
+
+```tf
+variant: flatcar
+version: 1.0.0
+...
+```
+
+### AWS
+
+* [Refresh](https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html) instances in autoscaling group when launch configuration changes ([#1208](https://github.com/poseidon/typhoon/pull/1208)) ([docs](https://typhoon.psdn.io/topics/maintenance/#node-configuration-updates), **important**)
+  * Worker launch configuration changes start an autoscaling group instance refresh to replace instances
+  * Instance refresh creates surge instances, waits for a warm-up period, then deletes old instances
+  * Changing `worker_type`, `disk_*`, `worker_price`, `worker_target_groups`, or Butane `worker_snippets` on existing worker nodes will replace instances
+  * New AMIs or changing `os_stream` will be ignored, to allow Fedora CoreOS or Flatcar Linux to keep themselves updated
+  * Previously, new launch configurations were made in the same way, but not applied to instances unless manually replaced
+* Rename worker autoscaling group `${cluster_name}-worker` ([#1202](https://github.com/poseidon/typhoon/pull/1202))
+  * Rename launch configuration `${cluster_name}-worker` instead of a random id
+
+### Google
+
+* [Roll](https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups) instance template changes to worker managed instance groups ([#1207](https://github.com/poseidon/typhoon/pull/1207)) ([docs](https://typhoon.psdn.io/topics/maintenance/#node-configuration-updates), **important**)
+  * Worker instance template changes roll out by gradually replacing instances
+  * Automatic rollouts create surge instances, wait for health checks, then delete old instances (0 unavailable instances)
+  * Changing `worker_type`, `disk_size`, `worker_preemptible`, or Butane `worker_snippets` on existing worker nodes will replace instances
+  * New compute images or changing `os_stream` will be ignored, to allow Fedora CoreOS or Flatcar Linux to keep themselves updated
+  * Previously, new instance templates were made in the same way, but not applied to instances unless manually replaced
+* Add health checks to worker managed instance groups (i.e. "autohealing") ([#1207](https://github.com/poseidon/typhoon/pull/1207))
+  * Use health checks to probe kube-proxy every 30s
+  * Replace worker nodes that fail the health check 6 times (3min)
+* Name `kube-apiserver` and `worker` health checks consistently ([#1207](https://github.com/poseidon/typhoon/pull/1207))
+  * Use name `${cluster_name}-apiserver-health` and `${cluster_name}-worker-health`
+* Rename managed instance group from `${cluster_name}-worker-group` to `${cluster_name}-worker` ([#1207](https://github.com/poseidon/typhoon/pull/1207))
+* Fix bug provisioning clusters with multiple controller nodes ([#1195](https://github.com/poseidon/typhoon/pull/1195))
+
+### Addons
+
+* Update Prometheus from v2.37.0 to [v2.38.0](https://github.com/prometheus/prometheus/releases/tag/v2.38.0)
+* Update Grafana from v9.0.3 to [v9.1.0](https://github.com/grafana/grafana/releases/tag/v9.1.0)
+
+## v1.24.3
+
+* Kubernetes [v1.24.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1243)
+* Update Cilium from v1.11.6 to [v1.11.7](https://github.com/cilium/cilium/releases/tag/v1.11.7)
+
+### Addons
+
+* Update ingress-nginx from v1.2.1 to [v1.3.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.3.0)
+* Update Prometheus from v2.36.1 to [v2.37.0](https://github.com/prometheus/prometheus/releases/tag/v2.37.0)
+* Update Grafana from v8.5.6 to [v9.0.3](https://github.com/grafana/grafana/releases/tag/v9.0.3)
+
+### Notes
+
+* Poseidon repos will soon change their default branch from `master` to `main`
+
+## v1.24.2
+
+* Kubernetes [v1.24.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1242)
+* Update Cilium from v1.11.5 to [v1.11.6](https://github.com/cilium/cilium/releases/tag/v1.11.6)
+* Update Calico from v3.22.2 to [v3.23.1](https://github.com/projectcalico/calico/releases/tag/v3.23.1)
+
+### Addons
+
+* Update Prometheus from v2.36.0 to [v2.36.1](https://github.com/prometheus/prometheus/releases/tag/v2.36.1)
+* Update Grafana from v8.5.3 to [v8.5.6](https://github.com/grafana/grafana/releases/tag/v8.5.6)
+* Update kube-state-metrics from v2.4.2 to [v2.5.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.5.0)
+
+## Known Issues
+
+* Skip AWS Terraform provider v4.17.0 to v4.19.0, which had a regression affecting workers joining ([#1173](https://github.com/poseidon/typhoon/issues/1173))
+
+## v1.24.1
+
+* Kubernetes [v1.24.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1241)
+* Update Cilium from v1.11.4 to [v1.11.5](https://github.com/cilium/cilium/releases/tag/v1.11.5)
+
+### Addons
+
+* Update Prometheus from v2.35.0 to [v2.36.0](https://github.com/prometheus/prometheus/releases/tag/v2.36.0)
+* Update Grafana from v8.5.1 to [v8.5.3](https://github.com/grafana/grafana/releases/tag/v8.5.3)
+* Update nginx-ingress from v1.2.0 to [v1.2.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.1)
+
+## v1.24.0
+
+* Kubernetes [v1.24.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#v1240)
+* Update etcd from v3.5.2 to [v3.5.4](https://github.com/etcd-io/etcd/releases/tag/v3.5.4)
+* Add Kubelet mounts to enable relabeling workload volumes ([#1152](https://github.com/poseidon/typhoon/pull/1152))
+  * StorageClass no longer require explicit SELinux mount contexts
+
+### Addons
+
+* Update nginx-ingress from v1.1.3 to [v1.2.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0)
+* Update Prometheus from v2.34.0 to [v2.35.0](https://github.com/prometheus/prometheus/releases/tag/v2.35.0)
+* Update Grafana from v8.4.5 to [v8.5.1](https://github.com/grafana/grafana/releases/tag/v8.5.1)
+
+## v1.23.6
+
+* Kubernetes [v1.23.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1236)
+* Update Cilium from v1.11.2 to [v1.11.4](https://github.com/cilium/cilium/releases/tag/v1.11.4)
+* Rename Cilium DaemonSet from `cilium-agent` to `cilium` to match Cilium CLI tools ([#303](https://github.com/poseidon/terraform-render-bootstrap/pull/303))
+* Update Calico from v3.22.1 to [v3.22.2](https://github.com/projectcalico/calico/releases/tag/v3.22.2)
+* Mount /etc/machine-id from host into Kubelet ([#1143](https://github.com/poseidon/typhoon/pull/1143))
+* Remove deprecated use of `key_algorithm` in `hashicorp/tls` resources
+
+### Azure
+
+* Allow upgrading Azure Terraform provider to v3.x ([#1144](https://github.com/poseidon/typhoon/pull/1144))
+* Rename `worker_address_prefix` output to `worker_address_prefixes`
+
+### Google Cloud
+
+* Fix issue on Flatcar Linux with controller nodes not ignoring os image changes ([#1149](https://github.com/poseidon/typhoon/pull/1149))
+  * Nodes will auto-update, Terraform should not attempt to delete/recreate them
+
+### Addons
+
+* Update nginx-ingress from v1.1.2 to [v1.1.3](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.3)
+* Update Prometheus from v2.33.5 to [v2.34.0](https://github.com/prometheus/prometheus/releases/tag/v2.34.0)
+* Update Grafana from v8.4.4 to [v8.4.5](https://github.com/grafana/grafana/releases/tag/v8.4.5)
+
+## v1.23.5
+
+* Kubernetes [v1.23.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1235)
+* Update Cilium from v1.11.1 to [v1.11.2](https://github.com/cilium/cilium/releases/tag/v1.11.2)
+* Update Calico from v3.21.2 to [v3.22.1](https://github.com/projectcalico/calico/releases/tag/v3.22.1)
+  * Fix [calico#5011](https://github.com/projectcalico/calico/issues/5011), broken since v1.23.0
+
+### Addons
+
+* Refresh Prometheus rules and Grafana dashboards ([#1136](https://github.com/poseidon/typhoon/pull/1136))
+* Update nginx-ingress from v1.1.1 to [v1.1.2](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.2)
+* Update Prometheus from v2.33.3 to [v2.33.5](https://github.com/prometheus/prometheus/releases/tag/v2.33.5)
+* Update Grafana from v8.4.1 to [v8.4.3](https://github.com/grafana/grafana/releases/tag/v8.4.3)
+* Update kube-state-metrics from v2.3.0 to [v2.4.2](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.4.2)
+
+## v1.23.4
+
+* Kubernetes [v1.23.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1234)
+* Update etcd from v3.5.1 to [v3.5.2](https://github.com/etcd-io/etcd/releases/tag/v3.5.2)
+* Change default CNI `networking` provider from `calico` to `cilium` ([#1114](https://github.com/poseidon/typhoon/pull/1114))
+
+### AWS
+
+* Allow upgrading AWS Terraform Provider to v4.x
+
+### Addons
+
+* Align nginx-ingress `--controller-class` with `IngressClass`
+  * Watch only `public` IngressClass objects, better [example](https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/)
+* Update Prometheus from v2.32.1 to [v2.33.3](https://github.com/prometheus/prometheus/releases/tag/v2.33.3)
+* Update Grafana from v8.3.6 to [v8.4.1](https://github.com/grafana/grafana/releases/tag/v8.4.1)
+
+## V1.23.3
+
+* Kubernetes [v1.23.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1233)
+
+### Flatcar Linux
+
+#### Google Cloud
+
+* Switch to using official Kinvolk Flatcar Linux images
+* Promote Typhoon on Flatcar Linux / Google Cloud to stable
+* Change `os_image` to `flatcar-stable`, `flatcar-beta`, or `flatcar-alpha` (**action required**)
+
+## v1.23.2
+
+* Kubernetes [v1.23.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1232)
+* Update Cilium from v1.11.0 to [v1.11.1](https://github.com/cilium/cilium/releases/tag/v1.11.1)
+* Remove Kubelet flag `--network-plugin`. Unused since `docker-shim` isn't used ([#1106](https://github.com/poseidon/typhoon/pull/1106))
+
+### Fedora CoreOS
+
+* Switch Kubernetes Container Runtime from `docker` to `containerd` ([#1101](https://github.com/poseidon/typhoon/pull/1101))
+* Mask `docker.service` to prevent it from being socket activated ([#1105](https://github.com/poseidon/typhoon/pull/1105))
+
+### Flatcar Linux
+
+#### AWS
+
+* Add experimental Flatcar Linux ARM64 support ([docs](https://typhoon.psdn.io/advanced/arm64/), [#1102](https://github.com/poseidon/typhoon/pull/1102))
+  * Add `arch` variable to AWS `kubernetes` and `workers` modules
+  * Allow arm64 full-cluster or mixed/hybrid cluster with arm64 workers
+  * Requires `flannel` or `cilium` CNI provider
+
+### DigitalOcean
+
+* Upgrade DigitalOcean Terraform provider to [v2.x](https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs) ([#1109](https://github.com/poseidon/typhoon/pull/1109))
+
+### Addons
+
+* Update nginx-ingress from v1.1.0 to [v1.1.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.1)
+* Update Grafana from v8.3.3 to [v8.3.4](https://github.com/grafana/grafana/releases/tag/v8.3.4)
+
+## v1.23.1
+
+* Kubernetes [v1.23.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1231)
+* Workaround Terraform v1.1 regression in `file` provisioner ([#1093](https://github.com/poseidon/typhoon/pull/1093))
+
+### Flatcar Linux
+
+* Switch Kubernetes Container Runtime from `docker` to `containerd` ([#1087](https://github.com/poseidon/typhoon/pull/1087))
+
+### Addons
+
+* Configure Prometheus to allow a custom scrape query parameter ([#1095](https://github.com/poseidon/typhoon/pull/1095))
+* Configure Prometheus to probe Kubernetes Ingress via `blackbox-exporter` ([#1096](https://github.com/poseidon/typhoon/pull/1096))
+* Fix Prometheus Service probes to use `blackbox-exporter`, not `blackbox` ([#1096](https://github.com/poseidon/typhoon/pull/1096))
+
+## v1.23.0
+
+* Kubernetes [v1.23.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#v1230)
+* Normalize CA cert mounts in static Pods and kube-proxy ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+* Set Kubelet resolver config to `/run/systemd/resolve/resolv.conf` ([#1082](https://github.com/poseidon/typhoon/pull/1082))
+* Update Cilium from v1.10.5 to [v1.11.0](https://github.com/cilium/cilium/releases/tag/v1.11.0) ([#1083](https://github.com/poseidon/typhoon/pull/1083))
+* With Calico, add missing `caliconodestatuses` CRD ([#289](https://github.com/poseidon/terraform-render-bootstrap/pull/289))
+* Change `enable_aggregation` default to true ([#279](https://github.com/poseidon/terraform-render-bootstrap/pull/279))
+* Remove deprecated `--port` from `kube-scheduler` ([#1078](https://github.com/poseidon/typhoon/pull/1078))
+
+### AWS
+
+* Change controller node default `disk_iops` to 3000 ([#1073](https://github.com/poseidon/typhoon/pull/1073))
+
+### Azure
+
+* Fix warning about deprecated `backend_address_pool_id` ([#1086](https://github.com/poseidon/typhoon/pull/1086))
+
+### Fedora CoreOS
+
+* Fix Fedora ARM64 workers to official Fedora CoreOS AMIs ([#1072](https://github.com/poseidon/typhoon/pull/1072))
+  * Should have been changed alongside controller AMIs in ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+  * Old Poseidon built ARM64 AMIs have been deleted
+
+### Addons
+
+* Update nginx-ingress from v1.0.5 to [v1.1.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.1.0)
+* Update Prometheus from v2.31.1 to [v2.32.0](https://github.com/prometheus/prometheus/releases/tag/v2.32.0)
+* Update kube-state-metrics from v2.2.4 to [v2.3.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.3.0)
+* Update node-exporter from v1.3.0 to [v1.3.1](https://github.com/prometheus/node_exporter/releases/tag/v1.3.1)
+* Update Grafana from v8.2.4 to [v8.3.3](https://github.com/grafana/grafana/releases/tag/v8.3.3)
+
+### Known Issues
+
+* Calico does not yet support Kubernetes v1.23.0, use `flannel` or `cilium` ([calico#5011](https://github.com/projectcalico/calico/issues/5011))
+
+## v1.22.4
+
+* Kubernetes [v1.22.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1224)
+* Update CoreDNS from v1.8.4 to [v1.8.6](https://github.com/poseidon/terraform-render-bootstrap/pull/284)
+* Update Calico from v3.20.2 to [v3.21.0](https://github.com/projectcalico/calico/releases/tag/v3.21.0)
+* Update flannel from v0.14.0 to [v0.15.1](https://github.com/flannel-io/flannel/releases/tag/v0.15.1)
+
+### Google
+
+* Allow use of Terraform provider `google` [v4.0+](https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.0.0)
+
+### Flatcar Linux
+
+* Change Kubelet mounts for cgroups v2 ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+* Update cgroup driver from cgroupfs to systemd (Flatcar Linux changed default) ([#1064](https://github.com/poseidon/typhoon/pull/1064))
+
+### Addons
+
+* Update Prometheus from v2.30.3 to [v2.31.1](https://github.com/prometheus/prometheus/releases/tag/v2.31.1)
+* Update node-exporter from v1.2.2 to [v1.3.0](https://github.com/prometheus/node_exporter/releases/tag/v1.3.0)
+* Update kube-state-metrics from v2.2.3 to [v2.2.4](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.4)
+* Update Grafana from v8.2.1 to [v8.2.4](https://github.com/grafana/grafana/releases/tag/v8.2.4)
+* Update nginx-ingress from v1.0.4 to [v1.0.5](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.5)
+
+## v1.23.3
+
+* Kubernetes [v1.22.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1223)
+* Update etcd from v3.5.0 to [v3.5.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.1)
+* Update Cilium from v1.10.4 to [v1.10.5](https://github.com/cilium/cilium/releases/tag/v1.10.5)
+* Update Calico from v3.20.1 to [v3.20.2](https://github.com/projectcalico/calico/releases/tag/v3.20.2)
+  * Use Calico's iptables legacy vs nft auto-detection
+* Update flannel from v0.13.0 to v0.14.0
+
+### Bare-Metal
+
+* Require Terraform provider `poseidon/matchbox` v0.5+ ([#1048](https://github.com/poseidon/typhoon/pull/1048))
+
+### Addons
+
+* Update nginx-ingress from v1.0.0 to [v1.0.4](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.4)
+* Update Prometheus from v2.29.2 to [v2.30.3](https://github.com/prometheus/prometheus/releases/tag/v2.30.3)
+* Update kube-state-metrics from v2.2.0 to [v2.2.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.3)
+* Update Grafana from v8.1.2 to [v8.2.1](https://github.com/grafana/grafana/releases/tag/v8.2.1)
+
+## v1.22.2
+
+* Kubernetes [v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1222)
+* Update Cilium from v1.10.3 to [v1.10.4](https://github.com/cilium/cilium/releases/tag/v1.10.4)
+* Update Calico from v3.20.0 to [v3.20.1](https://github.com/projectcalico/calico/releases/tag/v3.20.1)
+* Fix access to ClusterIP services with Cilium ([#276](https://github.com/poseidon/terraform-render-bootstrap/pull/276))
+
+### Fedora CoreOS
+
+* Use Fedora CoreOS ARM64 AMIs ([#1038](https://github.com/poseidon/typhoon/pull/1038))
+
+### Addons
+
+* Update Prometheus from v2.29.1 to [v2.29.2](https://github.com/prometheus/prometheus/releases/tag/v2.29.2)
+* Update kube-state-metrics from v2.1.1 to [v2.2.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.0)
+
+## v1.22.1
+
+* Kubernetes [v1.22.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1221)
+* Update Calico from v3.19.1 to [v3.20.0](https://github.com/projectcalico/calico/releases/tag/v3.20.0)
+
+### Addons
+
+* Update nginx-ingress from v1.0.0-beta.1 to [v1.0.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0)
+* Update Prometheus from v2.28.1 to [v2.29.1](https://github.com/prometheus/prometheus/releases/tag/v2.29.1)
+* Update Grafana from v8.1.1 to [v8.1.2](https://github.com/grafana/grafana/releases/tag/v8.1.2)
+
+## v1.22.0
+
+* Kubernetes [v1.22.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#v1220)
+* Update etcd from v3.4.16 to [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0)
+* Switch `kube-controller-manager` and `kube-scheduler` to use secure port only
+  * Update Prometheus config to discover endpoints and use a bearer token to scrape
+
+### Fedora CoreOS
+
+* Add Cilium cgroups v2 support on Fedora CoreOS
+* Update Butane Config version from v1.2.0 to v1.4.0
+  * Rename Fedora CoreOS Config to Butane Config
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.4.0
+
+### Addons
+
+* Update nginx-ingress from v0.47.0 to [v1.0.0-beta.1](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0-beta.1)
+* Update node-exporter from v1.2.0 to [v1.2.2](https://github.com/prometheus/node_exporter/releases/tag/v1.2.2)
+* Update kube-state-metrics from v2.1.0 to [v2.1.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.1)
+* Update Grafana from v8.0.6 to [v8.1.1](https://github.com/grafana/grafana/releases/tag/v8.1.1)
+
+## v1.21.3
+
+* Kubernetes [v1.21.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1213)
+* Update Cilium from v1.10.1 to [v1.10.3](https://github.com/cilium/cilium/releases/tag/v1.10.3)
+* Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.9+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+
+### AWS
+
+* Change default disk type from `gp2` to `gp3` ([#1012](https://github.com/poseidon/typhoon/pull/1012))
+
+### Addons
+
+* Update Prometheus from v2.28.0 to [v2.28.1](https://github.com/prometheus/prometheus/releases/tag/v2.28.1)
+* Update node-exporter from v1.1.2 to [v1.2.0](https://github.com/prometheus/node_exporter/releases/tag/v1.2.0)
+* Update Grafana from v8.0.3 to [v8.0.6](https://github.com/grafana/grafana/releases/tag/v8.0.6)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
+## v1.21.2
+
+* Kubernetes [v1.21.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1212)
+* Add Terraform v1.0.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x, v0.14.4+, and v0.15.x
+* Update CoreDNS from v1.8.0 to [v1.8.4]([#1006](https://github.com/poseidon/typhoon/pull/1006))
+* Update Cilium from v1.9.6 to [v1.10.1](https://github.com/cilium/cilium/releases/tag/v1.10.1)
+* Update Calico from v3.19.0 to [v3.19.1](https://github.com/projectcalico/calico/releases/tag/v3.19.1)
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0 to [v2.1.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.1.0)
+* Update Prometheus from v2.27.0 to [v2.28.0](https://github.com/prometheus/prometheus/releases/tag/v2.28.0)
+* Update Grafana from v7.5.6 to [v8.0.3](https://github.com/grafana/grafana/releases/tag/v8.0.3)
+* Update nginx-ingress from v0.46.0 to [v0.47.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.47.0)
+
+### Fedora CoreOS
+
+#### AWS
+
+* Extend experimental Fedora CoreOS arm64 support with Cilium
+  * CNI provider may now be `flannel` or `cilium` (new)
+
+#### Bare-Metal
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+#### DigitalOcean
+
+* Workaround systemd path unit issue [fedora-coreos-tracker/#861](https://github.com/coreos/fedora-coreos-tracker/issues/861)
+
+### Known Issues
+
+* Cilium with recent Fedora CoreOS will have networking issues ([fedora-coreos#881](https://github.com/coreos/fedora-coreos-tracker/issues/881)) (fixed in v1.21.4)
+
+## v1.21.1
+
+* Kubernetes [v1.21.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211)
+* Add Terraform v0.15.x support ([#974](https://github.com/poseidon/typhoon/pull/974))
+  * Continue to support Terraform v0.13.x and v0.14.4+
+* Update etcd from v3.4.15 to [v3.4.16](https://github.com/etcd-io/etcd/releases/tag/v3.4.16)
+* Update Cilium from v1.9.5 to [v1.9.6](https://github.com/cilium/cilium/releases/tag/v1.9.6)
+* Update Calico from v3.18.1 to [v3.19.0](https://github.com/projectcalico/calico/releases/tag/v3.19.0)
+
+### AWS
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Azure
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Google Cloud
+
+* Reduce the default `disk_size` from 40GB to 30GB ([#983](https://github.com/poseidon/typhoon/pull/983))
+
+### Fedora CoreOS
+
+* Update Kubelet mounts for cgroups v2 ([#978](https://github.com/poseidon/typhoon/pull/978))
+
+### Addons
+
+* Update kube-state-metrics from v2.0.0-rc.1 to [v2.0.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0)
+* Update Prometheus from v2.25.2 to [v2.27.0](https://github.com/prometheus/prometheus/releases/tag/v2.27.0)
+* Update Grafana from v7.5.3 to [v7.5.6](https://github.com/grafana/grafana/releases/tag/v7.5.6)
+* Update nginx-ingress from v0.45.0 to [v0.46.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.46.0)
+
+## v1.21.0
+
+* Kubernetes [v1.21.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1210)
+  * Enable `tokencleaner` controller ([#969](https://github.com/poseidon/typhoon/pull/969))
+  * Enable `kube-scheduler` and `kube-controller-manager` separate authn/z kubeconfig
+  * Change CNI config location from /etc/kubernetes/cni/net.d to /etc/cni/net.d ([#965](https://github.com/poseidon/typhoon/pull/965))
+  * Change `kube-controller-manager` to mount `/var/lib/kubelet/volumeplugins` directly
+  * Remove unused `cloud-provider` flags
+* Update Fedora CoreOS Config version from v1.1.0 to v1.2.0 ([#970](https://github.com/poseidon/typhoon/pull/970))
+  * Require [poseidon/ct](https://github.com/poseidon/terraform-provider-ct) Terraform provider v0.8+ ([notes](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.2.0
+
+### AWS
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Azure
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+* Remove deprecated `azurerm_lb_backend_address_pool` field `resource_group_name` ([#972](https://github.com/poseidon/typhoon/pull/972))
+
+### Google Cloud
+
+* Allow setting custom initial node taints on worker pools ([#968](https://github.com/poseidon/typhoon/pull/968))
+  * Add `node_taints` variable to internal `workers` pool module to set initial node taints
+  * Add `daemonset_tolerations` so `kube-system` DaemonSets can tolerate custom taints
+
+### Addons
+
+* Update nginx-ingress from v0.44.0 to [v0.45.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.45.0)
+* Update kube-state-metrics from v2.0.0-rc.0 to [v2.0.0-rc.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-rc.1)
+* Update Grafana from v7.4.5 to [v7.5.3](https://github.com/grafana/grafana/releases/tag/v7.5.3)
+
+## v1.20.5
+
+* Kubernetes [v1.20.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1205)
+* Update etcd from v3.4.14 to [v3.4.15](https://github.com/etcd-io/etcd/releases/tag/v3.4.15)
+* Update Cilium from v1.9.4 to [v1.9.5](https://github.com/cilium/cilium/releases/tag/v1.9.5)
+* Update Calico from v3.17.3 to [v3.18.1](https://github.com/projectcalico/calico/releases/tag/v3.18.1)
+* Update CoreDNS from v1.7.0 to [v1.8.0](https://coredns.io/2020/10/22/coredns-1.8.0-release/)
+* Mark bootstrap token as sensitive in Terraform plans ([#949](https://github.com/poseidon/typhoon/pull/949))
+
+### Fedora CoreOS
+
+* Set Kubelet `provider-id` ([#951](https://github.com/poseidon/typhoon/pull/951))
+
+### Flatcar Linux
+
+#### AWS
+
+* Set Kubelet `provider-id` ([#951](https://github.com/poseidon/typhoon/pull/951))
+* Remove `os_image` option `flatcar-edge` ([#943](https://github.com/poseidon/typhoon/pull/943))
+
+#### Azure
+
+* Remove `os_image` option `flatcar-edge` ([#943](https://github.com/poseidon/typhoon/pull/943))
+
+#### Bare-Metal
+
+* Remove `os_channel` option `flatcar-edge` ([#943](https://github.com/poseidon/typhoon/pull/943))
+
+### Addons
+
+* Update Prometheus from v2.25.0 to [v2.25.2](https://github.com/prometheus/prometheus/releases/tag/v2.25.2)
+* Update kube-state-metrics from v2.0.0-alpha.3 to [v2.0.0-rc.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-rc.0)
+  * Switch image from `quay.io` to `k8s.gcr.io` ([#946](https://github.com/poseidon/typhoon/pull/946))
+* Update node-exporter from v1.1.1 to [v1.1.2](https://github.com/prometheus/node_exporter/releases/tag/v1.1.2)
+* Update Grafana from v7.4.2 to [v7.4.5](https://github.com/grafana/grafana/releases/tag/v7.4.5)
+
+## v1.20.4
+
+* Kubernetes [v1.20.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1204)
+* Update Cilium from v1.9.1 to [v1.9.4](https://github.com/cilium/cilium/releases/tag/v1.9.4)
+* Update Calico from v3.17.1 to [v3.17.3](https://github.com/projectcalico/calico/releases/tag/v3.17.3)
+* Update flannel-cni from v0.4.1 to [v0.4.2](https://github.com/poseidon/flannel-cni/releases/tag/v0.4.2)
+
+### Addons
+
+* Update nginx-ingress from v0.43.0 to [v0.44.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.44.0)
+* Update Prometheus from v2.24.0 to [v2.25.0](https://github.com/prometheus/prometheus/releases/tag/v2.25.0)
+  * Update node-exporter from v1.0.1 to [v1.1.1](https://github.com/prometheus/node_exporter/releases/tag/v1.1.1)
+* Update Grafana from v7.3.7 to [v7.4.2](https://github.com/grafana/grafana/releases/tag/v7.4.2)
+
+## v1.20.2
+
+* Kubernetes [v1.20.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1202)
+* Support Terraform v0.13.x and v0.14.4+ ([#924](https://github.com/poseidon/typhoon/pull/923))
+
+### Addons
+
+* Update nginx-ingress from v0.41.2 to [v0.43.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.43.0)
+* Update Prometheus from v2.23.0 to [v2.24.0](https://github.com/prometheus/prometheus/releases/tag/v2.24.0)
+* Update Grafana from v7.3.6 to [v7.3.7](https://github.com/grafana/grafana/releases/tag/v7.3.7)
+
+## v1.20.1
+
+* Kubernetes [v1.20.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1201)
+
+### Fedora CoreOS
+
+* Fedora CoreOS 33 has stronger crypto defaults ([**notice**](https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_why_does_ssh_stop_working_after_upgrading_to_fedora_33), [#915](https://github.com/poseidon/typhoon/issues/915))
+  * Use a non-RSA SSH key or add the workaround provided in upstream [Fedora docs](https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_why_does_ssh_stop_working_after_upgrading_to_fedora_33) as a [snippet](https://typhoon.psdn.io/advanced/customization/#fedora-coreos) (**action required**)
+
+### Addons
+
+* Update Grafana from v7.3.5 to [v7.3.6](https://github.com/grafana/grafana/releases/tag/v7.3.6)
+
+## v1.20.0
+
+* Kubernetes [v1.20.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#v1200)
+* Add input variable validations ([#880](https://github.com/poseidon/typhoon/pull/880))
+  * Require Terraform v0.13+ ([migration guide](https://typhoon.psdn.io/topics/maintenance/#terraform-versions))
+* Set output sensitive to suppress console display for some cases ([#885](https://github.com/poseidon/typhoon/pull/885))
+* Add service account token [volume projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection) ([#897](https://github.com/poseidon/typhoon/pull/897))
+* Scope kube-scheduler and kube-controller-manager permissions ([#898](https://github.com/poseidon/typhoon/pull/898))
+* Update etcd from v3.4.12 to [v3.4.14](https://github.com/etcd-io/etcd/releases/tag/v3.4.14)
+* Update Calico from v3.16.5 to v3.17.1 ([#890](https://github.com/poseidon/typhoon/pull/890))
+  * Enable Calico MTU auto-detection
+  * Remove [workaround](https://github.com/poseidon/typhoon/pull/724) to Calico cni-plugin [issue](https://github.com/projectcalico/cni-plugin/issues/874)
+* Update Cilium from v1.9.0 to [v1.9.1](https://github.com/cilium/cilium/releases/tag/v1.9.1)
+* Relax `terraform-provider-ct` version constraint to v0.6+ ([#893](https://github.com/poseidon/typhoon/pull/893))
+  * Allow upgrading `terraform-provider-ct` to v0.7.x ([warn](https://typhoon.psdn.io/topics/maintenance/#upgrade-terraform-provider-ct))
+
+### AWS
+
+* Enable Network Load Balancer (NLB) dualstack ([#883](https://github.com/poseidon/typhoon/pull/883))
+  * NLB subnets assigned both IPv4 and IPv6 addresses
+  * NLB DNS name has both A and AAAA records
+  * NLB to target node traffic is IPv4 (no change)
+
+### Bare-Metal
+
+* Remove iSCSI `/etc/iscsi` and `iscsadm` mounts from Kubelet ([#912](https://github.com/poseidon/typhoon/pull/912))
+
+### Fedora CoreOS
+
+#### AWS
+
+* Fix AMI query for which could fail in some regions ([#887](https://github.com/poseidon/typhoon/pull/887))
+
+#### Bare-Metal
+
+* Promote Fedora CoreOS to stable
+* Use initramfs and rootfs images as initrd's ([#889](https://github.com/poseidon/typhoon/pull/889))
+  * Requires Fedora CoreOS version with rootfs images (e.g. 32.20200923.3.0+)
+
+### Addons
+
+* Update Prometheus from v2.22.2 to [v2.23.0](https://github.com/prometheus/prometheus/releases/tag/v2.23.0)
+* Update kube-state-metrics from v2.0.0-alpha.2 to [v2.0.0-alpha.3](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.3)
+* Update Grafana from v7.3.2 to [v7.3.5](https://github.com/grafana/grafana/releases/tag/v7.3.5)
+
+## v1.19.4
+
+* Kubernetes [v1.19.4](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1194)
+* Update Cilium from v1.8.4 to [v1.9.0](https://github.com/cilium/cilium/releases/tag/v1.9.0)
+* Update Calico from v3.16.3 to [v3.16.5](https://github.com/projectcalico/calico/releases/tag/v3.16.5)
+* Remove `asset_dir` variable (defaulted off in [v1.17.0](https://github.com/poseidon/typhoon/pull/595), deprecated in [v1.18.0](https://github.com/poseidon/typhoon/pull/678))
+
+### Fedora CoreOS
+
+* Improve `etcd-member.service` systemd unit ([#868](https://github.com/poseidon/typhoon/pull/868))
+  * Allow a snippet with a systemd dropin to set an alternate image (e.g. mirror)
+* Fix local node delete oneshot on node shutdown ([#856](https://github.com/poseidon/typhoon/pull/855))
+
+#### AWS
+
+* Add experimental Fedora CoreOS arm64 support ([docs](https://typhoon.psdn.io/advanced/arm64/), [#875](https://github.com/poseidon/typhoon/pull/875))
+  * Allow arm64 full-cluster or mixed/hybrid cluster with worker pools
+  * Add `arch` variable to cluster module
+  * Add `daemonset_tolerations` variable to cluster module
+  * Add `node_taints` variable to workers module
+  * Requires flannel CNI provider and use of experimental AMI (see docs)
+
+### Flatcar Linux
+
+* Rename `container-linux` modules to `flatcar-linux` ([#858](https://github.com/poseidon/typhoon/issues/858)) (**action required**)
+* Change on-host system containers from rkt to docker
+  * Change `etcd-member.service` container runnner from rkt to docker ([#867](https://github.com/poseidon/typhoon/pull/867))
+  * Change `kubelet.service` container runner from rkt-fly to docker ([#855](https://github.com/poseidon/typhoon/pull/855))
+  * Change `bootstrap.service` container runner from rkt to docker ([#873](https://github.com/poseidon/typhoon/pull/873))
+  * Change `delete-node.service` to use docker and an inline ExecStart ([#855](https://github.com/poseidon/typhoon/pull/855))
+* Fix local node delete oneshot on node shutdown ([#855](https://github.com/poseidon/typhoon/pull/855))
+* Remove CoreOS Container Linux Matchbox profiles ([#859](https://github.com/poseidon/typhoon/pull/858))
+
+### Addons
+
+* Update nginx-ingress from v0.40.2 to [v0.41.2](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.41.2)
+* Update Prometheus from v2.22.0 to [v2.22.1](https://github.com/prometheus/prometheus/releases/tag/v2.22.1)
+* Update kube-state-metrics from v2.0.0-alpha.1 to [v2.0.0-alpha.2](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.2)
+* Update Grafana from v7.2.1 to [v7.3.2](https://github.com/grafana/grafana/releases/tag/v7.3.2)
+
+## v1.19.3
+
+* Kubernetes [v1.19.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1193)
+* Update Cilium from v1.8.3 to [v1.8.4](https://github.com/cilium/cilium/releases/tag/v1.8.4)
+* Update Calico from v1.15.3 to [v1.16.3](https://github.com/projectcalico/calico/releases/tag/v3.16.3) ([#851](https://github.com/poseidon/typhoon/pull/851))
+* Update flannel from v0.13.0-rc2 to v0.13.0 ([#219](https://github.com/poseidon/terraform-render-bootstrap/pull/219))
+
+### Flatcar Linux
+
+* Remove references to CoreOS Container Linux ([#839](https://github.com/poseidon/typhoon/pull/839))
+  * Fix error querying for coreos AMI on AWS ([#838](https://github.com/poseidon/typhoon/issues/838))
+
+### Addons
+
+* Update nginx-ingress from v0.35.0 to [v0.40.2](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.40.2)
+* Update Grafana from v7.1.5 to [v7.2.1](https://github.com/grafana/grafana/releases/tag/v7.2.1)
+* Update Prometheus from v2.21.0 to [v2.22.0](https://github.com/prometheus/prometheus/releases/tag/v2.22.0)
+  * Update kube-state-metrics from v1.9.7 to [v2.0.0-alpha.1](https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.0.0-alpha.1)
+
+## v1.19.2
+
+* Kubernetes [v1.19.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1192)
+* Update flannel from v0.12.0 to v0.13.0-rc2 ([#216](https://github.com/poseidon/terraform-render-bootstrap/pull/216))
+  * Update flannel-cni from v0.4.0 to v0.4.1
+  * Update CNI plugins from v0.8.6 to v0.8.7
+
+### Addons
+
+* Refresh Prometheus rules/alerts and Grafana dashboards ([#831](https://github.com/poseidon/typhoon/pull/831))
+* Reduce apiserver metrics cardinality for non-core APIs ([#830](https://github.com/poseidon/typhoon/pull/830))
+
+## v1.19.1
+
+* Kubernetes [v1.19.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1191)
+  * Change control plane seccomp annotations to GA `seccompProfile` ([#822](https://github.com/poseidon/typhoon/pull/822))
+* Update Cilium from v1.8.2 to [v1.8.3](https://github.com/cilium/cilium/releases/tag/v1.8.3)
+  * Promote Cilium from experimental to general availability ([#827](https://github.com/poseidon/typhoon/pull/827))
+* Update Calico from v1.15.2 to [v1.15.3](https://github.com/projectcalico/calico/releases/tag/v3.15.3)
+
+### Fedora CoreOS
+
+* Update Fedora CoreOS Config version from v1.0.0 to v1.1.0
+  * Require any [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customizations to update to v1.1.0
+
+### Addons
+
+* Update IngressClass resources to `networking.k8s.io/v1` ([#824](https://github.com/poseidon/typhoon/pull/824))
+* Update Prometheus from v2.20.0 to [v2.21.0](https://github.com/prometheus/prometheus/releases/tag/v2.21.0)
+  * Remove Kubernetes node name labelmap `relabel_config` from etcd, Kubelet, and CAdvisor scrape config ([#828](https://github.com/poseidon/typhoon/pull/828))
+
+## v1.19.0
+
+* Kubernetes [v1.19.0](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1190)
+* Update etcd from v3.4.10 to [v3.4.12](https://github.com/etcd-io/etcd/releases/tag/v3.4.12)
+* Update Calico from v3.15.1 to [v3.15.2](https://docs.projectcalico.org/v3.15/release-notes/)
+
+### Fedora CoreOS
+
+* Fix race condition during bootstrap of multi-controller clusters ([#808](https://github.com/poseidon/typhoon/pull/808))
+  * Fix SELinux label of bootstrap-secrets on non-bootstrap controllers
+
+### Addons
+
+* Introduce [fleetlock](https://github.com/poseidon/fleetlock) for Fedora CoreOS reboot coordination ([#814](https://github.com/poseidon/typhoon/pull/814))
+* Update nginx-ingress from v0.34.1 to [v0.35.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v0.35.0)
+  * Repository changed to `k8s.gcr.io/ingress-nginx/controller`
+* Update Grafana from v7.1.3 to [v7.1.5](https://github.com/grafana/grafana/releases/tag/v7.1.5)
+
+## v1.18.8
+
+* Kubernetes [v1.18.8](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1188)
+* Migrate from Terraform v0.12.x to v0.13.x ([#804](https://github.com/poseidon/typhoon/pull/804)) (**action required**)
+  * Recommend Terraform v0.13.x ([migration guide](https://typhoon.psdn.io/topics/maintenance/#terraform-versions))
+  * Support automatic install of poseidon's provider plugins ([poseidon/ct](https://registry.terraform.io/providers/poseidon/ct/latest), [poseidon/matchbox](https://registry.terraform.io/providers/poseidon/matchbox/latest))
+  * Require Terraform v0.12.26+ (migration compatibility)
+  * Require `terraform-provider-ct` v0.6.1
+  * Require `terraform-provider-matchbox` v0.4.1
+* Update etcd from v3.4.9 to [v3.4.10](https://github.com/etcd-io/etcd/releases/tag/v3.4.10)
+* Update CoreDNS from v1.6.7 to [v1.7.0](https://coredns.io/2020/06/15/coredns-1.7.0-release/)
+* Update Cilium from v1.8.1 to [v1.8.2](https://github.com/cilium/cilium/releases/tag/v1.8.2)
+* Update [coreos/flannel-cni](https://github.com/coreos/flannel-cni) to [poseidon/flannel-cni](https://github.com/poseidon/flannel-cni) ([#798](https://github.com/poseidon/typhoon/pull/798))
+  * Update CNI plugins and fix CVEs with Flannel CNI (non-default)
+  * Transition to a poseidon maintained container image
+
+### AWS
+
+* Allow `terraform-provider-aws` v3.0+ ([#803](https://github.com/poseidon/typhoon/pull/803))
+  * Recommend updating `terraform-provider-aws` to v3.0+
+  * Continue to allow v2.23+, no v3.x specific features are used
+
+### DigitalOcean
+
+* Require `terraform-provider-digitalocean` v1.21+ for Terraform v0.13.x (unenforced)
+* Require `terraform-provider-digitalocean` v1.20+ for Terraform v0.12.x
+
+### Fedora CoreOS
+
+* Fix support for Flannel with Fedora CoreOS ([#795](https://github.com/poseidon/typhoon/pull/795))
+  * Configure `flannel.1` link to select its own MAC address to solve flannel
+  pod-to-pod traffic drops starting with default link changes in Fedora CoreOS
+  32.20200629.3.0 ([details](https://github.com/coreos/fedora-coreos-tracker/issues/574#issuecomment-665487296))
+
+#### Addons
+
+* Update Prometheus from v2.19.2 to [v2.20.0](https://github.com/prometheus/prometheus/releases/tag/v2.20.0)
+* Update Grafana from v7.0.6 to [v7.1.3](https://github.com/grafana/grafana/releases/tag/v7.1.3)
+
 ## v1.18.6

 * Kubernetes [v1.18.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1186)
--- a/README.md
+++ b/README.md
@ -1,4 +1,6 @@
-# Typhoon <img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">
+# Typhoon [![Release](https://img.shields.io/github/v/release/poseidon/typhoon)](https://github.com/poseidon/typhoon/releases) [![Stars](https://img.shields.io/github/stars/poseidon/typhoon)](https://github.com/poseidon/typhoon/stargazers) [![Sponsors](https://img.shields.io/github/sponsors/poseidon?logo=github)](https://github.com/sponsors/poseidon) [![Mastodon](https://img.shields.io/badge/follow-news-6364ff?logo=mastodon)](https://fosstodon.org/@typhoon)
+
+<img align="right" src="https://storage.googleapis.com/poseidon/typhoon-logo.png">

 Typhoon is a minimal and free Kubernetes distribution.

@ -11,10 +13,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.6 (upstream)
+* Kubernetes v1.26.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/cl/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [preemptible](https://typhoon.psdn.io/flatcar-linux/google-cloud/#preemption) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, or other [addons](https://typhoon.psdn.io/addons/overview/)

 ## Modules
@ -27,19 +29,28 @@ Typhoon is available for [Fedora CoreOS](https://getfedora.org/coreos/).
 |---------------|------------------|------------------|--------|
 | AWS           | Fedora CoreOS | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | stable |
 | Azure         | Fedora CoreOS | [azure/fedora-coreos/kubernetes](azure/fedora-coreos/kubernetes) | alpha |
-| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | beta |
+| Bare-Metal    | Fedora CoreOS | [bare-metal/fedora-coreos/kubernetes](bare-metal/fedora-coreos/kubernetes) | stable |
 | DigitalOcean  | Fedora CoreOS | [digital-ocean/fedora-coreos/kubernetes](digital-ocean/fedora-coreos/kubernetes) | beta |
 | Google Cloud  | Fedora CoreOS | [google-cloud/fedora-coreos/kubernetes](google-cloud/fedora-coreos/kubernetes) | stable |

+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Fedora CoreOS (ARM64) | [aws/fedora-coreos/kubernetes](aws/fedora-coreos/kubernetes) | alpha |
+
 Typhoon is available for [Flatcar Linux](https://www.flatcar-linux.org/releases/).

 | Platform      | Operating System | Terraform Module | Status |
 |---------------|------------------|------------------|--------|
-| AWS           | Flatcar Linux    | [aws/container-linux/kubernetes](aws/container-linux/kubernetes) | stable |
-| Azure         | Flatcar Linux    | [azure/container-linux/kubernetes](azure/container-linux/kubernetes) | alpha |
-| Bare-Metal    | Flatcar Linux    | [bare-metal/container-linux/kubernetes](bare-metal/container-linux/kubernetes) | stable |
-| DigitalOcean | Flatcar Linux  | [digital-ocean/container-linux/kubernetes](digital-ocean/container-linux/kubernetes) | beta |
-| Google Cloud  | Flatcar Linux  | [google-cloud/container-linux/kubernetes](google-cloud/container-linux/kubernetes) | beta |
+| AWS           | Flatcar Linux    | [aws/flatcar-linux/kubernetes](aws/flatcar-linux/kubernetes) | stable |
+| Azure         | Flatcar Linux    | [azure/flatcar-linux/kubernetes](azure/flatcar-linux/kubernetes) | alpha |
+| Bare-Metal    | Flatcar Linux    | [bare-metal/flatcar-linux/kubernetes](bare-metal/flatcar-linux/kubernetes) | stable |
+| DigitalOcean | Flatcar Linux  | [digital-ocean/flatcar-linux/kubernetes](digital-ocean/flatcar-linux/kubernetes) | beta |
+| Google Cloud  | Flatcar Linux  | [google-cloud/flatcar-linux/kubernetes](google-cloud/flatcar-linux/kubernetes) | stable |
+
+| Platform      | Operating System | Terraform Module | Status |
+|---------------|------------------|------------------|--------|
+| AWS           | Flatcar Linux (ARM64) | [aws/flatcar-linux/kubernetes](aws/flatcar-linux/kubernetes) | alpha |
+| Azure         | Flatcar Linux (ARM64) | [azure/flatcar-linux/kubernetes](azure/flatcar-linux/kubernetes) | alpha |

 ## Documentation

@ -54,7 +65,7 @@ Define a Kubernetes cluster by using the Terraform module for your chosen platfo

 ```tf
 module "yavin" {
-  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.18.6"
+  source = "git::https://github.com/poseidon/typhoon//google-cloud/fedora-coreos/kubernetes?ref=v1.26.1"

  # Google Cloud
  cluster_name  = "yavin"
@ -63,7 +74,7 @@ module "yavin" {
  dns_zone_name = "example-zone"

  # configuration
-  ssh_authorized_key = "ssh-rsa AAAAB3Nz..."
+  ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."

  # optional
  worker_count = 2
@ -93,9 +104,9 @@ In 4-8 minutes (varies by platform), the cluster will be ready. This Google Clou
 $ export KUBECONFIG=/home/user/.kube/configs/yavin-config
 $ kubectl get nodes
 NAME                                       ROLES    STATUS  AGE  VERSION
-yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.18.6
-yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.18.6
-yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.18.6
+yavin-controller-0.c.example-com.internal  <none>   Ready   6m   v1.26.1
+yavin-worker-jrbf.c.example-com.internal   <none>   Ready   5m   v1.26.1
+yavin-worker-mzdm.c.example-com.internal   <none>   Ready   5m   v1.26.1
 ```

 List the pods.
@ -126,7 +137,7 @@ Typhoon is strict about minimalism, maturity, and scope. These are not in scope:

 ## Help

-Ask questions on the IRC #typhoon channel on [freenode.net](http://freenode.net/).
+Schedule a meeting via [Github Sponsors](https://github.com/sponsors/poseidon?frequency=one-time) to discuss your use case.

 ## Motivation

@ -136,12 +147,24 @@ Typhoon addresses real world needs, which you may share. It is honest about limi

 ## Social Contract

-Typhoon is not a product, trial, or free-tier. It is not run by a company, does not offer support or services, and does not accept or make any money. It is not associated with any operating system or platform vendor.
+Typhoon is not a product, trial, or free-tier. Typhoon does not offer support, services, or charge money. And Typhoon is independent of operating system or platform vendors.

 Typhoon clusters will contain only [free](https://www.debian.org/intro/free) components. Cluster components will not collect data on users without their permission.

-## Donations
+## Sponsors

-Typhoon does not accept money donations. Instead, we encourage you to donate to one of [these organizations](https://github.com/poseidon/typhoon/wiki/Donations) to show your appreciation.
+Poseidon's Github [Sponsors](https://github.com/sponsors/poseidon) support the infrastructure and operational costs of providing Typhoon.

-* [DigitalOcean](https://www.digitalocean.com/) kindly provides credits to support Typhoon test clusters.
+<a href="https://www.digitalocean.com/">
+    <img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_blue.svg" width="201px">
+</a>
+<br>
+<br>
+
+<a href="https://deploy.equinix.com/">
+  <img src="https://storage.googleapis.com/poseidon/equinix.png" width="201px">
+</a>
+<br>
+<br>
+
+If you'd like your company here, please contact dghubble at psdn.io.
--- a/addons/grafana/dashboards-coredns.yaml
+++ b/addons/grafana/dashboards-coredns.yaml
@ -37,6 +37,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -49,6 +50,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -72,7 +74,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m])) by (proto)",
+                  "expr": "sum(rate(coredns_dns_requests_total{instance=~\"$instance\"}[5m])) by (proto)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{proto}}",
@ -128,6 +130,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -140,6 +143,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -163,7 +167,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(coredns_dns_request_type_count_total{instance=~\"$instance\"}[5m])) by (type)",
+                  "expr": "sum(rate(coredns_dns_requests_total{instance=~\"$instance\"}[5m])) by (type)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{type}}",
@ -219,6 +223,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -231,6 +236,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -254,7 +260,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m])) by (zone)",
+                  "expr": "sum(rate(coredns_dns_requests_total{instance=~\"$instance\"}[5m])) by (zone)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{zone}}",
@ -323,6 +329,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -335,6 +342,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -428,6 +436,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -440,6 +449,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -463,7 +473,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(coredns_dns_response_rcode_count_total{instance=~\"$instance\"}[5m])) by (rcode)",
+                  "expr": "sum(rate(coredns_dns_responses_total{instance=~\"$instance\"}[5m])) by (rcode)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{rcode}}",
@ -532,6 +542,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -544,6 +555,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -637,6 +649,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -649,6 +662,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -755,6 +769,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -767,6 +782,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -790,7 +806,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(coredns_cache_size{instance=~\"$instance\"}) by (type)",
+                  "expr": "sum(coredns_cache_entries{instance=~\"$instance\"}) by (type)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{type}}",
@ -846,6 +862,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -858,6 +875,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": false
              },
--- a/addons/grafana/dashboards-etcd.yaml
+++ b/addons/grafana/dashboards-etcd.yaml
@ -11,7 +11,6 @@ data:
      "editable": true,
      "gnetId": null,
      "hideControls": false,
-      "id": 6,
      "links": [

      ],
@ -141,7 +140,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(grpc_server_started_total{job=\"$cluster\",grpc_type=\"unary\"}[5m]))",
+                  "expr": "sum(rate(grpc_server_started_total{job=\"$cluster\",grpc_type=\"unary\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "RPC Rate",
@ -150,7 +149,7 @@ data:
                  "step": 2
                },
                {
-                  "expr": "sum(rate(grpc_server_handled_total{job=\"$cluster\",grpc_type=\"unary\",grpc_code!=\"OK\"}[5m]))",
+                  "expr": "sum(rate(grpc_server_handled_total{job=\"$cluster\",grpc_type=\"unary\",grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "RPC Failed Rate",
@ -343,7 +342,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "etcd_debugging_mvcc_db_total_size_in_bytes{job=\"$cluster\"}",
+                  "expr": "etcd_mvcc_db_total_size_in_bytes{job=\"$cluster\"}",
                  "hide": false,
                  "interval": "",
                  "intervalFactor": 2,
@ -431,7 +430,7 @@ data:
              "steppedLine": true,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\"$cluster\"}[$__rate_interval])) by (instance, le))",
                  "hide": false,
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} WAL fsync",
@ -440,7 +439,7 @@ data:
                  "step": 4
                },
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\"$cluster\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\"$cluster\"}[$__rate_interval])) by (instance, le))",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} DB fsync",
                  "metric": "etcd_disk_backend_commit_duration_seconds_bucket",
@ -618,7 +617,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(etcd_network_client_grpc_received_bytes_total{job=\"$cluster\"}[5m])",
+                  "expr": "rate(etcd_network_client_grpc_received_bytes_total{job=\"$cluster\"}[$__rate_interval])",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} Client Traffic In",
                  "metric": "etcd_network_client_grpc_received_bytes_total",
@ -704,7 +703,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(etcd_network_client_grpc_sent_bytes_total{job=\"$cluster\"}[5m])",
+                  "expr": "rate(etcd_network_client_grpc_sent_bytes_total{job=\"$cluster\"}[$__rate_interval])",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} Client Traffic Out",
                  "metric": "etcd_network_client_grpc_sent_bytes_total",
@ -790,7 +789,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(etcd_network_peer_received_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
+                  "expr": "sum(rate(etcd_network_peer_received_bytes_total{job=\"$cluster\"}[$__rate_interval])) by (instance)",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} Peer Traffic In",
                  "metric": "etcd_network_peer_received_bytes_total",
@ -879,7 +878,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(etcd_network_peer_sent_bytes_total{job=\"$cluster\"}[5m])) by (instance)",
+                  "expr": "sum(rate(etcd_network_peer_sent_bytes_total{job=\"$cluster\"}[$__rate_interval])) by (instance)",
                  "hide": false,
                  "interval": "",
                  "intervalFactor": 2,
@ -973,7 +972,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(etcd_server_proposals_failed_total{job=\"$cluster\"}[5m]))",
+                  "expr": "sum(rate(etcd_server_proposals_failed_total{job=\"$cluster\"}[$__rate_interval]))",
                  "intervalFactor": 2,
                  "legendFormat": "Proposal Failure Rate",
                  "metric": "etcd_server_proposals_failed_total",
@ -989,7 +988,7 @@ data:
                  "step": 2
                },
                {
-                  "expr": "sum(rate(etcd_server_proposals_committed_total{job=\"$cluster\"}[5m]))",
+                  "expr": "sum(rate(etcd_server_proposals_committed_total{job=\"$cluster\"}[$__rate_interval]))",
                  "intervalFactor": 2,
                  "legendFormat": "Proposal Commit Rate",
                  "metric": "etcd_server_proposals_committed_total",
@ -997,7 +996,7 @@ data:
                  "step": 2
                },
                {
-                  "expr": "sum(rate(etcd_server_proposals_applied_total{job=\"$cluster\"}[5m]))",
+                  "expr": "sum(rate(etcd_server_proposals_applied_total{job=\"$cluster\"}[$__rate_interval]))",
                  "intervalFactor": 2,
                  "legendFormat": "Proposal Apply Rate",
                  "refId": "D",
@ -1132,6 +1131,131 @@ data:
                  "show": true
                }
              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": 0,
+              "editable": true,
+              "error": false,
+              "fieldConfig": {
+                "defaults": {
+                  "custom": {
+
+                  }
+                },
+                "overrides": [
+
+                ]
+              },
+              "fill": 0,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 7,
+                "w": 12,
+                "x": 0,
+                "y": 28
+              },
+              "hiddenSeries": false,
+              "id": 42,
+              "isNew": true,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": false,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "nullPointMode": "connected",
+              "options": {
+                "alertThreshold": true
+              },
+              "percentage": false,
+              "pluginVersion": "7.4.3",
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\"$cluster\"}[$__rate_interval])))",
+                  "interval": "",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{instance}} Peer round trip time",
+                  "metric": "etcd_network_peer_round_trip_time_seconds_bucket",
+                  "refId": "A",
+                  "step": 2
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeRegions": [
+
+              ],
+              "timeShift": null,
+              "title": "Peer round trip time",
+              "tooltip": {
+                "msResolution": false,
+                "shared": true,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "$$hashKey": "object:925",
+                  "decimals": null,
+                  "format": "s",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "$$hashKey": "object:926",
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ],
+              "yaxis": {
+                "align": false,
+                "alignLevel": null
+              }
            }
          ],
          "title": "New row"
@ -1141,7 +1265,7 @@ data:
      "sharedCrosshair": false,
      "style": "dark",
      "tags": [
-
+        "etcd-mixin"
      ],
      "templating": {
        "list": [
@ -1151,7 +1275,7 @@ data:
              "value": "Prometheus"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -1177,7 +1301,7 @@ data:

            ],
            "query": "label_values(etcd_server_has_leader, job)",
-            "refresh": 1,
+            "refresh": 2,
            "regex": "",
            "sort": 2,
            "tagValuesQuery": "",
--- a/addons/grafana/dashboards-k8s-network.yaml
+++ b/addons/grafana/dashboards-k8s-network.yaml
@ -0,0 +1,7644 @@
+apiVersion: v1
+data:
+  cluster-total.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "id": 3,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "id": 4,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "columns": [
+            {
+              "text": "Time",
+              "value": "Time"
+            },
+            {
+              "text": "Value #A",
+              "value": "Value #A"
+            },
+            {
+              "text": "Value #B",
+              "value": "Value #B"
+            },
+            {
+              "text": "Value #C",
+              "value": "Value #C"
+            },
+            {
+              "text": "Value #D",
+              "value": "Value #D"
+            },
+            {
+              "text": "Value #E",
+              "value": "Value #E"
+            },
+            {
+              "text": "Value #F",
+              "value": "Value #F"
+            },
+            {
+              "text": "Value #G",
+              "value": "Value #G"
+            },
+            {
+              "text": "Value #H",
+              "value": "Value #H"
+            },
+            {
+              "text": "namespace",
+              "value": "namespace"
+            }
+          ],
+          "datasource": "$datasource",
+          "fill": 1,
+          "fontSize": "90%",
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null as zero",
+          "renderer": "flot",
+          "scroll": true,
+          "showHeader": true,
+          "sort": {
+            "col": 0,
+            "desc": false
+          },
+          "spaceLength": 10,
+          "span": 24,
+          "styles": [
+            {
+              "alias": "Time",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Time",
+              "thresholds": [
+
+              ],
+              "type": "hidden",
+              "unit": "short"
+            },
+            {
+              "alias": "Current Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #A",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Current Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #B",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #C",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #D",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Rate of Received Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #E",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #F",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Received Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #G",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #H",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Namespace",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": true,
+              "linkTooltip": "Drill down",
+              "linkUrl": "d/8b7a8b326d7a6f1f04244066368c67af/kubernetes-networking-namespace-pods?orgId=1&refresh=30s&var-namespace=$__cell",
+              "pattern": "namespace",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "short"
+            }
+          ],
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "B",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "C",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "D",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "E",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "F",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "G",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "H",
+              "step": 10
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Status",
+          "type": "table"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 6,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 11
+              },
+              "id": 7,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Received",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 11
+              },
+              "id": 8,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Transmitted",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Average Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 11
+          },
+          "id": 9,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth History",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 12
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": true,
+            "avg": true,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": true,
+            "min": true,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "legend": {
+            "alignAsTable": true,
+            "avg": true,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": true,
+            "min": true,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{namespace}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 30
+          },
+          "id": 12,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 31
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 40
+              },
+              "id": 14,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 31
+          },
+          "id": 15,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 50
+              },
+              "id": 16,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 59
+              },
+              "id": 17,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\".+\"}[$interval:$resolution])) by (namespace))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{namespace}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 59
+              },
+              "id": 18,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+                {
+                  "targetBlank": true,
+                  "title": "What is TCP Retransmit?",
+                  "url": "https://accedian.com/enterprises/blog/network-packet-loss-retransmissions-and-duplicate-acknowledgements/"
+                }
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(rate(node_netstat_Tcp_RetransSegs{cluster=\"$cluster\"}[$interval:$resolution]) / rate(node_netstat_Tcp_OutSegs{cluster=\"$cluster\"}[$interval:$resolution])) by (instance))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{instance}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of TCP Retransmits out of all sent segments",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 24,
+                "x": 0,
+                "y": 59
+              },
+              "id": 19,
+              "legend": {
+                "alignAsTable": true,
+                "avg": true,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": true,
+                "min": true,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": true
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+                {
+                  "targetBlank": true,
+                  "title": "Why monitor SYN retransmits?",
+                  "url": "https://github.com/prometheus/node_exporter/issues/1023#issuecomment-408128365"
+                }
+              ],
+              "minSpan": 24,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(rate(node_netstat_TcpExt_TCPSynRetrans{cluster=\"$cluster\"}[$interval:$resolution]) / rate(node_netstat_Tcp_RetransSegs{cluster=\"$cluster\"}[$interval:$resolution])) by (instance))",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{instance}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of TCP SYN Retransmits out of all retransmits",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Cluster",
+      "uid": "ff635a025bcfea7bc3dd4f508990a3e9",
+      "version": 0
+    }
+  namespace-by-pod.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "height": 9,
+          "id": 3,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "height": 9,
+          "id": 4,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "columns": [
+            {
+              "text": "Time",
+              "value": "Time"
+            },
+            {
+              "text": "Value #A",
+              "value": "Value #A"
+            },
+            {
+              "text": "Value #B",
+              "value": "Value #B"
+            },
+            {
+              "text": "Value #C",
+              "value": "Value #C"
+            },
+            {
+              "text": "Value #D",
+              "value": "Value #D"
+            },
+            {
+              "text": "Value #E",
+              "value": "Value #E"
+            },
+            {
+              "text": "Value #F",
+              "value": "Value #F"
+            },
+            {
+              "text": "pod",
+              "value": "pod"
+            }
+          ],
+          "datasource": "$datasource",
+          "fill": 1,
+          "fontSize": "100%",
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null as zero",
+          "renderer": "flot",
+          "scroll": true,
+          "showHeader": true,
+          "sort": {
+            "col": 0,
+            "desc": false
+          },
+          "spaceLength": 10,
+          "span": 24,
+          "styles": [
+            {
+              "alias": "Time",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Time",
+              "thresholds": [
+
+              ],
+              "type": "hidden",
+              "unit": "short"
+            },
+            {
+              "alias": "Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #A",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #B",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Rate of Received Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #C",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #D",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Received Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #E",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #F",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Pod",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": true,
+              "linkTooltip": "Drill down",
+              "linkUrl": "d/7a18067ce943a40ae25454675c19ff5c/kubernetes-networking-pod?orgId=1&refresh=30s&var-namespace=$namespace&var-pod=$__cell",
+              "pattern": "pod",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "short"
+            }
+          ],
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "B",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "C",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "D",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "E",
+              "step": 10
+            },
+            {
+              "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "F",
+              "step": 10
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Status",
+          "type": "table"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 19
+          },
+          "id": 6,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 20
+          },
+          "id": 7,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 20
+          },
+          "id": 8,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 29
+          },
+          "id": 9,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 30
+              },
+              "id": 10,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 30
+              },
+              "id": 11,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 30
+          },
+          "id": 12,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 40
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 40
+              },
+              "id": 14,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Namespace (Pods)",
+      "uid": "8b7a8b326d7a6f1f04244066368c67af",
+      "version": 0
+    }
+  namespace-by-workload.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "id": 3,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ workload }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "id": 4,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ workload }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "columns": [
+            {
+              "text": "Time",
+              "value": "Time"
+            },
+            {
+              "text": "Value #A",
+              "value": "Value #A"
+            },
+            {
+              "text": "Value #B",
+              "value": "Value #B"
+            },
+            {
+              "text": "Value #C",
+              "value": "Value #C"
+            },
+            {
+              "text": "Value #D",
+              "value": "Value #D"
+            },
+            {
+              "text": "Value #E",
+              "value": "Value #E"
+            },
+            {
+              "text": "Value #F",
+              "value": "Value #F"
+            },
+            {
+              "text": "Value #G",
+              "value": "Value #G"
+            },
+            {
+              "text": "Value #H",
+              "value": "Value #H"
+            },
+            {
+              "text": "workload",
+              "value": "workload"
+            }
+          ],
+          "datasource": "$datasource",
+          "fill": 1,
+          "fontSize": "90%",
+          "gridPos": {
+            "h": 9,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null as zero",
+          "renderer": "flot",
+          "scroll": true,
+          "showHeader": true,
+          "sort": {
+            "col": 0,
+            "desc": false
+          },
+          "spaceLength": 10,
+          "span": 24,
+          "styles": [
+            {
+              "alias": "Time",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Time",
+              "thresholds": [
+
+              ],
+              "type": "hidden",
+              "unit": "short"
+            },
+            {
+              "alias": "Current Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #A",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Current Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #B",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Received",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #C",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Average Bandwidth Transmitted",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #D",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "Bps"
+            },
+            {
+              "alias": "Rate of Received Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #E",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #F",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Received Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #G",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Rate of Transmitted Packets Dropped",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": false,
+              "linkTooltip": "Drill down",
+              "linkUrl": "",
+              "pattern": "Value #H",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "pps"
+            },
+            {
+              "alias": "Workload",
+              "colorMode": null,
+              "colors": [
+
+              ],
+              "dateFormat": "YYYY-MM-DD HH:mm:ss",
+              "decimals": 2,
+              "link": true,
+              "linkTooltip": "Drill down",
+              "linkUrl": "d/728bf77cc1166d2f3133bf25846876cc/kubernetes-networking-workload?orgId=1&refresh=30s&var-namespace=$namespace&var-type=$type&var-workload=$__cell",
+              "pattern": "workload",
+              "thresholds": [
+
+              ],
+              "type": "number",
+              "unit": "short"
+            }
+          ],
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "B",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "C",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "D",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "E",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "F",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "G",
+              "step": 10
+            },
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "table",
+              "instant": true,
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "H",
+              "step": 10
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Status",
+          "type": "table"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 19
+          },
+          "id": 6,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 20
+              },
+              "id": 7,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ workload }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Received",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 20
+              },
+              "id": 8,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ workload }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Transmitted",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Average Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 29
+          },
+          "id": 9,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth HIstory",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 38
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{workload}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 38
+          },
+          "id": 11,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{workload}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 39
+          },
+          "id": 12,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 40
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 40
+              },
+              "id": 14,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 40
+          },
+          "id": 15,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 41
+              },
+              "id": 16,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 41
+              },
+              "id": 17,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{workload}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "deployment",
+              "value": "deployment"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Namespace (Workload)",
+      "uid": "bbb2a765a623ae38130206c7d94a160f",
+      "version": 0
+    }
+  pod-total.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "height": 9,
+          "id": 3,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace: $pod",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "cacheTimeout": null,
+          "colorBackground": false,
+          "colorValue": false,
+          "colors": [
+            "#299c46",
+            "rgba(237, 129, 40, 0.89)",
+            "#d44a3a"
+          ],
+          "datasource": "$datasource",
+          "decimals": 0,
+          "format": "time_series",
+          "gauge": {
+            "maxValue": 100,
+            "minValue": 0,
+            "show": false,
+            "thresholdLabels": false,
+            "thresholdMarkers": true
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "height": 9,
+          "id": 4,
+          "interval": null,
+          "links": [
+
+          ],
+          "mappingType": 1,
+          "mappingTypes": [
+            {
+              "name": "value to text",
+              "value": 1
+            },
+            {
+              "name": "range to text",
+              "value": 2
+            }
+          ],
+          "maxDataPoints": 100,
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "nullText": null,
+          "options": {
+            "fieldOptions": {
+              "calcs": [
+                "last"
+              ],
+              "defaults": {
+                "max": 10000000000,
+                "min": 0,
+                "title": "$namespace: $pod",
+                "unit": "Bps"
+              },
+              "mappings": [
+
+              ],
+              "override": {
+
+              },
+              "thresholds": [
+                {
+                  "color": "dark-green",
+                  "index": 0,
+                  "value": null
+                },
+                {
+                  "color": "dark-yellow",
+                  "index": 1,
+                  "value": 5000000000
+                },
+                {
+                  "color": "dark-red",
+                  "index": 2,
+                  "value": 7000000000
+                }
+              ],
+              "values": false
+            }
+          },
+          "postfix": "",
+          "postfixFontSize": "50%",
+          "prefix": "",
+          "prefixFontSize": "50%",
+          "rangeMaps": [
+            {
+              "from": "null",
+              "text": "N/A",
+              "to": "null"
+            }
+          ],
+          "span": 12,
+          "sparkline": {
+            "fillColor": "rgba(31, 118, 189, 0.18)",
+            "full": false,
+            "lineColor": "rgb(31, 120, 193)",
+            "show": false
+          },
+          "tableColumn": "",
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution]))",
+              "format": "time_series",
+              "instant": null,
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "thresholds": "",
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "type": "gauge",
+          "valueFontSize": "80%",
+          "valueMaps": [
+            {
+              "op": "=",
+              "text": "N/A",
+              "value": "null"
+            }
+          ],
+          "valueName": "current"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 11
+          },
+          "id": 6,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 11
+          },
+          "id": 7,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 20
+          },
+          "id": 8,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 21
+              },
+              "id": 9,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 21
+              },
+              "id": 10,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 0,
+                "y": 32
+              },
+              "id": 12,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 10,
+                "w": 12,
+                "x": 12,
+                "y": 32
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\",namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "",
+              "value": ""
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}, pod)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "pod",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{cluster=\"$cluster\",namespace=~\"$namespace\"}, pod)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Pod",
+      "uid": "7a18067ce943a40ae25454675c19ff5c",
+      "version": 0
+    }
+  workload-total.json: |-
+    {
+      "__inputs": [
+
+      ],
+      "__requires": [
+
+      ],
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "hideControls": false,
+      "id": null,
+      "links": [
+
+      ],
+      "panels": [
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Current Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 1
+          },
+          "id": 3,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ pod }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Received",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": true,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 1
+          },
+          "id": 4,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "current",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": false,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "minSpan": 24,
+          "nullPointMode": "null",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 24,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{ pod }}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Current Rate of Bytes Transmitted",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "series",
+            "name": null,
+            "show": false,
+            "values": [
+              "current"
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 5,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 11
+              },
+              "id": 6,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ pod }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Received",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": true,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 11
+              },
+              "id": 7,
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": true,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "sort": "current",
+                "sortDesc": true,
+                "total": false,
+                "values": true
+              },
+              "lines": false,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "minSpan": 24,
+              "nullPointMode": "null",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 24,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{ pod }}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Average Rate of Bytes Transmitted",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "series",
+                "name": null,
+                "show": false,
+                "values": [
+                  "current"
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Average Bandwidth",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 11
+          },
+          "id": 8,
+          "panels": [
+
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Bandwidth HIstory",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 0,
+            "y": 12
+          },
+          "id": 9,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Receive Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 2,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 9,
+            "w": 12,
+            "x": 12,
+            "y": 12
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 2,
+          "links": [
+
+          ],
+          "minSpan": 12,
+          "nullPointMode": "connected",
+          "paceLength": 10,
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "span": 12,
+          "stack": true,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+              "format": "time_series",
+              "intervalFactor": 1,
+              "legendFormat": "{{pod}}",
+              "refId": "A",
+              "step": 10
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Transmit Bandwidth",
+          "tooltip": {
+            "shared": true,
+            "sort": 2,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            },
+            {
+              "format": "Bps",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": 0,
+              "show": true
+            }
+          ]
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 22
+              },
+              "id": 12,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 22
+              },
+              "id": 13,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Packets",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": true,
+          "collapsed": true,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 22
+          },
+          "id": 14,
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 0,
+                "y": 23
+              },
+              "id": 15,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Received Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 2,
+              "fillGradient": 0,
+              "gridPos": {
+                "h": 9,
+                "w": 12,
+                "x": 12,
+                "y": 23
+              },
+              "id": 16,
+              "legend": {
+                "alignAsTable": false,
+                "avg": false,
+                "current": false,
+                "hideEmpty": true,
+                "hideZero": true,
+                "max": false,
+                "min": false,
+                "rightSide": false,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 2,
+              "links": [
+
+              ],
+              "minSpan": 12,
+              "nullPointMode": "connected",
+              "paceLength": 10,
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 12,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\",namespace=~\"$namespace\"}[$interval:$resolution])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "format": "time_series",
+                  "intervalFactor": 1,
+                  "legendFormat": "{{pod}}",
+                  "refId": "A",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Rate of Transmitted Packets Dropped",
+              "tooltip": {
+                "shared": true,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Errors",
+          "titleSize": "h6",
+          "type": "row"
+        }
+      ],
+      "refresh": "10s",
+      "rows": [
+
+      ],
+      "schemaVersion": 18,
+      "style": "dark",
+      "tags": [
+        "kubernetes-mixin"
+      ],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "text": "default",
+              "value": "default"
+            },
+            "hide": 0,
+            "label": "Data Source",
+            "name": "datasource",
+            "options": [
+
+            ],
+            "query": "prometheus",
+            "refresh": 1,
+            "regex": "",
+            "type": "datasource"
+          },
+          {
+            "allValue": null,
+            "current": {
+
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(kube_pod_info{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": ".+",
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "kube-system",
+              "value": "kube-system"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\"}, namespace)",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "namespace",
+            "options": [
+
+            ],
+            "query": "label_values(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "",
+              "value": ""
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\"}, workload)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "workload",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\"}, workload)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "deployment",
+              "value": "deployment"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\"}, workload_type)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\",namespace=~\"$namespace\", workload=~\"$workload\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "resolution",
+            "options": [
+              {
+                "selected": false,
+                "text": "30s",
+                "value": "30s"
+              },
+              {
+                "selected": true,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              }
+            ],
+            "query": "30s,5m,1h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "5m",
+              "value": "5m"
+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "interval",
+            "options": [
+              {
+                "selected": true,
+                "text": "4h",
+                "value": "4h"
+              }
+            ],
+            "query": "4h",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "interval",
+            "useTags": false
+          }
+        ]
+      },
+      "time": {
+        "from": "now-1h",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "30d"
+        ]
+      },
+      "timezone": "UTC",
+      "title": "Kubernetes / Networking / Workload",
+      "uid": "728bf77cc1166d2f3133bf25846876cc",
+      "version": 0
+    }
+kind: ConfigMap
+metadata:
+  name: grafana-dashboards-k8s-network
+  namespace: monitoring
--- a/addons/grafana/dashboards-k8s-nodes.yaml
+++ b/addons/grafana/dashboards-k8s-nodes.yaml
@ -20,2349 +20,2106 @@ data:
      "id": null,
      "links": [

+      ],
+      "panels": [
+        {
+          "datasource": "$datasource",
+          "fieldConfig": {
+            "defaults": {
+              "links": [
+
+              ],
+              "mappings": [
+
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
+          "targets": [
+            {
+              "expr": "sum(kubelet_node_name{cluster=\"$cluster\", job=\"kubelet\"})",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Running Kubelets",
+          "transparent": false,
+          "type": "stat"
+        },
+        {
+          "datasource": "$datasource",
+          "fieldConfig": {
+            "defaults": {
+              "links": [
+
+              ],
+              "mappings": [
+
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 4,
+            "y": 0
+          },
+          "id": 3,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
+          "targets": [
+            {
+              "expr": "sum(kubelet_running_pods{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}) OR sum(kubelet_running_pod_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"})",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Running Pods",
+          "transparent": false,
+          "type": "stat"
+        },
+        {
+          "datasource": "$datasource",
+          "fieldConfig": {
+            "defaults": {
+              "links": [
+
+              ],
+              "mappings": [
+
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 8,
+            "y": 0
+          },
+          "id": 4,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
+          "targets": [
+            {
+              "expr": "sum(kubelet_running_containers{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}) OR sum(kubelet_running_container_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"})",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Running Containers",
+          "transparent": false,
+          "type": "stat"
+        },
+        {
+          "datasource": "$datasource",
+          "fieldConfig": {
+            "defaults": {
+              "links": [
+
+              ],
+              "mappings": [
+
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 12,
+            "y": 0
+          },
+          "id": 5,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
+          "targets": [
+            {
+              "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\", state=\"actual_state_of_world\"})",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Actual Volume Count",
+          "transparent": false,
+          "type": "stat"
+        },
+        {
+          "datasource": "$datasource",
+          "fieldConfig": {
+            "defaults": {
+              "links": [
+
+              ],
+              "mappings": [
+
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 16,
+            "y": 0
+          },
+          "id": 6,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
+          "targets": [
+            {
+              "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\",state=\"desired_state_of_world\"})",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Desired Volume Count",
+          "transparent": false,
+          "type": "stat"
+        },
+        {
+          "datasource": "$datasource",
+          "fieldConfig": {
+            "defaults": {
+              "links": [
+
+              ],
+              "mappings": [
+
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+
+                ]
+              },
+              "unit": "none"
+            }
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 4,
+            "x": 20,
+            "y": 0
+          },
+          "id": 7,
+          "links": [
+
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "7",
+          "targets": [
+            {
+              "expr": "sum(rate(kubelet_node_config_error{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval]))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Config Error Count",
+          "transparent": false,
+          "type": "stat"
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 7
+          },
+          "id": 8,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(kubelet_runtime_operations_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (operation_type, instance)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_type}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Operation Rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 7
+          },
+          "id": 9,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(kubelet_runtime_operations_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_type}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Operation Error Rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 14
+          },
+          "id": 10,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_runtime_operations_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_type}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Operation duration 99th quantile",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 21
+          },
+          "id": 11,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} pod",
+              "refId": "A"
+            },
+            {
+              "expr": "sum(rate(kubelet_pod_worker_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} worker",
+              "refId": "B"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Pod Start Rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 21
+          },
+          "id": 12,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} pod",
+              "refId": "A"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} worker",
+              "refId": "B"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Pod Start Duration",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 28
+          },
+          "id": 13,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(storage_operation_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_name, volume_plugin)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Storage Operation Rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 28
+          },
+          "id": 14,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(storage_operation_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_name, volume_plugin)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Storage Operation Error Rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 35
+          },
+          "id": 15,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(storage_operation_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_name, volume_plugin, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Storage Operation Duration 99th quantile",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 42
+          },
+          "id": 16,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(kubelet_cgroup_manager_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{operation_type}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Cgroup manager operation rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 42
+          },
+          "id": 17,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_cgroup_manager_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, operation_type, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{operation_type}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Cgroup manager 99th quantile",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "description": "Pod lifecycle event generator",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 49
+          },
+          "id": 18,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(kubelet_pleg_relist_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance)",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "PLEG relist rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 12,
+            "y": 49
+          },
+          "id": 19,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_interval_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "PLEG relist interval",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 56
+          },
+          "id": 20,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "PLEG relist duration",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 63
+          },
+          "id": 21,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "2xx",
+              "refId": "A"
+            },
+            {
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "3xx",
+              "refId": "B"
+            },
+            {
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "4xx",
+              "refId": "C"
+            },
+            {
+              "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "5xx",
+              "refId": "D"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "RPC Rate",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "ops",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 24,
+            "x": 0,
+            "y": 70
+          },
+          "id": 22,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "max": false,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\"}[$__rate_interval])) by (instance, verb, url, le))",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}} {{verb}} {{url}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Request duration 99th quantile",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "s",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 8,
+            "x": 0,
+            "y": 77
+          },
+          "id": 23,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "process_resident_memory_bytes{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Memory",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "bytes",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "bytes",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 8,
+            "x": 8,
+            "y": 77
+          },
+          "id": 24,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[$__rate_interval])",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "CPU usage",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "short",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "short",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        },
+        {
+          "aliasColors": {
+
+          },
+          "bars": false,
+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "$datasource",
+          "fill": 1,
+          "fillGradient": 0,
+          "gridPos": {
+            "h": 7,
+            "w": 8,
+            "x": 16,
+            "y": 77
+          },
+          "id": 25,
+          "legend": {
+            "alignAsTable": false,
+            "avg": false,
+            "current": false,
+            "max": false,
+            "min": false,
+            "rightSide": false,
+            "show": true,
+            "sideWidth": null,
+            "total": false,
+            "values": false
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [
+
+          ],
+          "nullPointMode": "null",
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "seriesOverrides": [
+
+          ],
+          "spaceLength": 10,
+          "stack": false,
+          "steppedLine": false,
+          "targets": [
+            {
+              "expr": "go_goroutines{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}",
+              "format": "time_series",
+              "intervalFactor": 2,
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "thresholds": [
+
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Goroutines",
+          "tooltip": {
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "type": "graph",
+          "xaxis": {
+            "buckets": null,
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": [
+
+            ]
+          },
+          "yaxes": [
+            {
+              "format": "short",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "format": "short",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ]
+        }
      ],
      "refresh": "10s",
      "rows": [
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {

-              },
-              "id": 2,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(up{cluster=\"$cluster\", job=\"kubelet\"})",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Up",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 3,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(kubelet_running_pod_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"})",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Running Pods",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 4,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(kubelet_running_container_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"})",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Running Container",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 5,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\", state=\"actual_state_of_world\"})",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Actual Volume Count",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 6,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\",state=\"desired_state_of_world\"})",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Desired Volume Count",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            },
-            {
-              "cacheTimeout": null,
-              "colorBackground": false,
-              "colorValue": false,
-              "colors": [
-                "#299c46",
-                "rgba(237, 129, 40, 0.89)",
-                "#d44a3a"
-              ],
-              "datasource": "$datasource",
-              "format": "none",
-              "gauge": {
-                "maxValue": 100,
-                "minValue": 0,
-                "show": false,
-                "thresholdLabels": false,
-                "thresholdMarkers": true
-              },
-              "gridPos": {
-
-              },
-              "id": 7,
-              "interval": null,
-              "links": [
-
-              ],
-              "mappingType": 1,
-              "mappingTypes": [
-                {
-                  "name": "value to text",
-                  "value": 1
-                },
-                {
-                  "name": "range to text",
-                  "value": 2
-                }
-              ],
-              "maxDataPoints": 100,
-              "nullPointMode": "connected",
-              "nullText": null,
-              "postfix": "",
-              "postfixFontSize": "50%",
-              "prefix": "",
-              "prefixFontSize": "50%",
-              "rangeMaps": [
-                {
-                  "from": "null",
-                  "text": "N/A",
-                  "to": "null"
-                }
-              ],
-              "span": 2,
-              "sparkline": {
-                "fillColor": "rgba(31, 118, 189, 0.18)",
-                "full": false,
-                "lineColor": "rgb(31, 120, 193)",
-                "show": false
-              },
-              "tableColumn": "",
-              "targets": [
-                {
-                  "expr": "sum(rate(kubelet_node_config_error{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": "",
-              "title": "Config Error Count",
-              "tooltip": {
-                "shared": false
-              },
-              "type": "singlestat",
-              "valueFontSize": "80%",
-              "valueMaps": [
-                {
-                  "op": "=",
-                  "text": "N/A",
-                  "value": "null"
-                }
-              ],
-              "valueName": "min"
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 8,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(kubelet_runtime_operations_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (operation_type, instance)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_type}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Operation Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 9,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(kubelet_runtime_operations_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_type)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_type}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Operation Error Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 10,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_runtime_operations_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_type, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_type}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Operation duration 99th quantile",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 11,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} pod",
-                  "refId": "A"
-                },
-                {
-                  "expr": "sum(rate(kubelet_pod_worker_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} worker",
-                  "refId": "B"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Pod Start Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 12,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} pod",
-                  "refId": "A"
-                },
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} worker",
-                  "refId": "B"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Pod Start Duration",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 13,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "hideEmpty": true,
-                "hideZero": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(storage_operation_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Storage Operation Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 14,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "hideEmpty": true,
-                "hideZero": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(storage_operation_errors_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Storage Operation Error Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 15,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "hideEmpty": true,
-                "hideZero": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(storage_operation_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Storage Operation Duration 99th quantile",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 16,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(kubelet_cgroup_manager_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, operation_type)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{operation_type}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Cgroup manager operation rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 17,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_cgroup_manager_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, operation_type, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{operation_type}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Cgroup manager 99th quantile",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "description": "Pod lifecycle event generator",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 18,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(kubelet_pleg_relist_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "PLEG relist rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 19,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 6,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_interval_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "PLEG relist interval",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 20,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])) by (instance, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "PLEG relist duration",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 21,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "2xx",
-                  "refId": "A"
-                },
-                {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "3xx",
-                  "refId": "B"
-                },
-                {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "4xx",
-                  "refId": "C"
-                },
-                {
-                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "5xx",
-                  "refId": "D"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "RPC Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 22,
-              "legend": {
-                "alignAsTable": true,
-                "avg": false,
-                "current": true,
-                "max": false,
-                "min": false,
-                "rightSide": true,
-                "show": true,
-                "total": false,
-                "values": true
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 12,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", instance=~\"$instance\"}[5m])) by (instance, verb, url, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{verb}} {{url}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Request duration 99th quantile",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 23,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Memory",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "bytes",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "bytes",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 24,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}[5m])",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "CPU usage",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 25,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "go_goroutines{cluster=\"$cluster\",job=\"kubelet\",instance=~\"$instance\"}",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "Goroutines",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": null,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        }
      ],
      "schemaVersion": 14,
      "style": "dark",
@ -2377,7 +2134,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -2401,7 +2158,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, cluster)",
+            "query": "label_values(up{job=\"kubelet\"}, cluster)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -2421,13 +2178,13 @@ data:
            "datasource": "$datasource",
            "hide": 0,
            "includeAll": true,
-            "label": null,
+            "label": "instance",
            "multi": false,
            "name": "instance",
            "options": [

            ],
-            "query": "label_values(kubelet_runtime_operations_total{cluster=\"$cluster\", job=\"kubelet\"}, instance)",
+            "query": "label_values(up{job=\"kubelet\",cluster=\"$cluster\"}, instance)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -2470,7 +2227,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Kubelet",
      "uid": "3138fa155d5915769fbded898ac09fd9",
      "version": 0
@ -2524,7 +2281,11 @@ data:

              },
              "id": 2,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
              "links": [

              ],
@ -2563,7 +2324,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(up{job=\"kube-proxy\"})",
+                  "expr": "sum(up{cluster=\"$cluster\", job=\"kube-proxy\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -2595,18 +2356,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 3,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -2630,7 +2394,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(kubeproxy_sync_proxy_rules_duration_seconds_count{job=\"kube-proxy\", instance=~\"$instance\"}[5m]))",
+                  "expr": "sum(rate(kubeproxy_sync_proxy_rules_duration_seconds_count{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "rate",
@ -2686,10 +2450,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 4,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -2698,6 +2464,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -2721,7 +2488,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99,rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket{job=\"kube-proxy\", instance=~\"$instance\"}[5m]))",
+                  "expr": "histogram_quantile(0.99,rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -2790,18 +2557,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 5,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -2825,7 +2595,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(kubeproxy_network_programming_duration_seconds_count{job=\"kube-proxy\", instance=~\"$instance\"}[5m]))",
+                  "expr": "sum(rate(kubeproxy_network_programming_duration_seconds_count{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "rate",
@ -2881,10 +2651,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 6,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -2893,6 +2665,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -2916,7 +2689,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(kubeproxy_network_programming_duration_seconds_bucket{job=\"kube-proxy\", instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(kubeproxy_network_programming_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\"}[$__rate_interval])) by (instance, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -2985,18 +2758,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 7,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3020,28 +2796,28 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "2xx",
                  "refId": "A"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "3xx",
                  "refId": "B"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "4xx",
                  "refId": "C"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-proxy\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "5xx",
@ -3097,18 +2873,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 8,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3132,7 +2911,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{job=\"kube-proxy\",instance=~\"$instance\",verb=\"POST\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\",verb=\"POST\"}[$__rate_interval])) by (verb, url, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{verb}} {{url}}",
@ -3201,10 +2980,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 9,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -3213,6 +2994,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -3236,7 +3018,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{job=\"kube-proxy\", instance=~\"$instance\", verb=\"GET\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-proxy\", instance=~\"$instance\", verb=\"GET\"}[$__rate_interval])) by (verb, url, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{verb}} {{url}}",
@ -3305,18 +3087,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 10,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3340,7 +3125,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "process_resident_memory_bytes{job=\"kube-proxy\",instance=~\"$instance\"}",
+                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -3396,18 +3181,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 11,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3431,7 +3219,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(process_cpu_seconds_total{job=\"kube-proxy\",instance=~\"$instance\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\"}[$__rate_interval])",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -3487,18 +3275,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 12,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3522,7 +3313,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "go_goroutines{job=\"kube-proxy\",instance=~\"$instance\"}",
+                  "expr": "go_goroutines{cluster=\"$cluster\", job=\"kube-proxy\",instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -3592,7 +3383,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -3606,6 +3397,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kube-proxy\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
@ -3616,7 +3433,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kubeproxy_network_programming_duration_seconds_bucket{job=\"kube-proxy\"}, instance)",
+            "query": "label_values(up{job=\"kube-proxy\", cluster=\"$cluster\", job=\"kube-proxy\"}, instance)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -3659,7 +3476,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Proxy",
      "uid": "632e265de029684c40b21cb76bca4f94",
      "version": 0
--- a/addons/grafana/dashboards-k8s-resources-1.yaml
+++ b/addons/grafana/dashboards-k8s-resources-1.yaml
@ -31,11 +31,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 1,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -59,7 +62,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\", cluster=\"$cluster\"}[$__interval]))",
+                  "expr": "cluster:node_cpu:ratio_rate5m{cluster=\"$cluster\"}",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -72,7 +75,7 @@ data:
              "title": "CPU Utilisation",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -115,11 +118,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 2,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -143,7 +149,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_cpu_cores{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"cpu\",cluster=\"$cluster\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -156,7 +162,7 @@ data:
              "title": "CPU Requests Commitment",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -199,11 +205,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 3,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -227,7 +236,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_cpu_cores{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"cpu\",cluster=\"$cluster\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -240,7 +249,7 @@ data:
              "title": "CPU Limits Commitment",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -283,11 +292,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 4,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -311,7 +323,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "1 - sum(:node_memory_MemAvailable_bytes:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_memory_bytes{cluster=\"$cluster\"})",
+                  "expr": "1 - sum(:node_memory_MemAvailable_bytes:sum{cluster=\"$cluster\"}) / sum(node_memory_MemTotal_bytes{job=\"node-exporter\",cluster=\"$cluster\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -324,7 +336,7 @@ data:
              "title": "Memory Utilisation",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -367,11 +379,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 5,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -395,7 +410,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_memory_bytes{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"memory\",cluster=\"$cluster\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -408,7 +423,7 @@ data:
              "title": "Memory Requests Commitment",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -451,11 +466,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 6,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -479,7 +497,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable_memory_bytes{cluster=\"$cluster\"})",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) / sum(kube_node_status_allocatable{job=\"kube-state-metrics\",resource=\"memory\",cluster=\"$cluster\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -492,7 +510,7 @@ data:
              "title": "Memory Limits Commitment",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -546,11 +564,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 7,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -574,7 +595,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -590,7 +611,7 @@ data:
              "title": "CPU Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -644,11 +665,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 8,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -686,8 +710,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 0,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                  "pattern": "Value #A",
                  "thresholds": [

@ -704,8 +729,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 0,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to workloads",
-                  "linkUrl": "./d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                  "pattern": "Value #B",
                  "thresholds": [

@ -722,6 +748,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -740,6 +767,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -758,6 +786,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -776,6 +805,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -794,6 +824,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #G",
@ -812,8 +843,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
                  "pattern": "namespace",
                  "thresholds": [

@ -839,7 +871,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(kube_pod_owner{job=\"kube-state-metrics\", cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -848,7 +880,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "count(avg(mixin_pod_workload{cluster=\"$cluster\"}) by (workload, namespace)) by (namespace)",
+                  "expr": "count(avg(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\"}) by (workload, namespace)) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -857,7 +889,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -866,7 +898,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -875,7 +907,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace) / sum(namespace_cpu:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -884,7 +916,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_cpu:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -893,7 +925,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\"}) by (namespace) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\"}) by (namespace) / sum(namespace_cpu:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -910,7 +942,7 @@ data:
              "title": "CPU Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -965,11 +997,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 9,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -993,7 +1028,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -1009,7 +1044,7 @@ data:
              "title": "Memory Usage (w/o cache)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1063,11 +1098,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 10,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1105,8 +1143,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 0,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                  "pattern": "Value #A",
                  "thresholds": [

@ -1123,8 +1162,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 0,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to workloads",
-                  "linkUrl": "./d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
+                  "linkUrl": "/d/a87fb0d919ec0ea5f6543124e16c42a5/k8s-resources-workloads-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell_1",
                  "pattern": "Value #B",
                  "thresholds": [

@ -1141,6 +1181,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -1159,6 +1200,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -1177,6 +1219,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -1195,6 +1238,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -1213,6 +1257,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #G",
@ -1231,8 +1276,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
                  "pattern": "namespace",
                  "thresholds": [

@ -1258,7 +1304,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(kube_pod_owner{job=\"kube-state-metrics\", cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1267,7 +1313,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "count(avg(mixin_pod_workload{cluster=\"$cluster\"}) by (workload, namespace)) by (namespace)",
+                  "expr": "count(avg(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\"}) by (workload, namespace)) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1276,7 +1322,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1285,7 +1331,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1294,7 +1340,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(namespace_memory:kube_pod_container_resource_requests:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1303,7 +1349,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(namespace_memory:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1312,7 +1358,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\"}) by (namespace)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", container!=\"\"}) by (namespace) / sum(namespace_memory:kube_pod_container_resource_limits:sum{cluster=\"$cluster\"}) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1329,7 +1375,7 @@ data:
              "title": "Requests by Namespace",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -1384,11 +1430,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 11,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1426,6 +1475,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -1444,6 +1494,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -1462,6 +1513,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -1480,6 +1532,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -1498,6 +1551,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -1516,6 +1570,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -1534,8 +1589,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
                  "pattern": "namespace",
                  "thresholds": [

@ -1561,7 +1617,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1570,7 +1626,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1579,7 +1635,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1588,7 +1644,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1597,7 +1653,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1606,7 +1662,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -1623,7 +1679,7 @@ data:
              "title": "Current Network Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -1661,7 +1717,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
          "titleSize": "h6"
        },
        {
@ -1678,11 +1734,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 12,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1701,12 +1760,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -1722,7 +1781,7 @@ data:
              "title": "Receive Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1753,19 +1812,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -1776,11 +1823,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 13,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1799,12 +1849,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -1820,7 +1870,7 @@ data:
              "title": "Transmit Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1857,7 +1907,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
          "titleSize": "h6"
        },
        {
@ -1874,11 +1924,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 14,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1897,12 +1950,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -1918,7 +1971,7 @@ data:
              "title": "Average Container Bandwidth by Namespace: Received",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1949,19 +2002,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -1972,11 +2013,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 15,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1995,12 +2039,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2016,7 +2060,7 @@ data:
              "title": "Average Container Bandwidth by Namespace: Transmitted",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2053,7 +2097,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Average Container Bandwidth by Namespace",
          "titleSize": "h6"
        },
        {
@ -2070,11 +2114,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 16,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2093,12 +2140,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2114,7 +2161,7 @@ data:
              "title": "Rate of Received Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2129,7 +2176,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -2145,19 +2192,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -2168,11 +2203,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 17,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2191,12 +2229,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2212,7 +2250,7 @@ data:
              "title": "Rate of Transmitted Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2227,7 +2265,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -2249,7 +2287,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
          "titleSize": "h6"
        },
        {
@ -2266,11 +2304,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 18,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2289,12 +2330,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2310,7 +2351,7 @@ data:
              "title": "Rate of Received Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2325,7 +2366,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -2341,19 +2382,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -2364,11 +2393,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 19,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2387,12 +2419,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\".+\"}[$__interval])) by (namespace)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=~\".+\"}[$__rate_interval])) by (namespace)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{namespace}}",
@ -2408,7 +2440,198 @@ data:
              "title": "Rate of Transmitted Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Rate of Packets Dropped",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 20,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(namespace) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{namespace}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS(Reads+Writes)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 21,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{namespace}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut(Read+Write)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2445,7 +2668,315 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Storage IO",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 22,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "sort": {
+                "col": 4,
+                "desc": true
+              },
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "IOPS(Reads)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Reads + Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Throughput(Read)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Read + Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Namespace",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/85a562078cdf77779eaa1add43ccec1e/k8s-resources-namespace?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$__cell",
+                  "pattern": "namespace",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(namespace) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace!=\"\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Current Storage IO",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution",
          "titleSize": "h6"
        }
      ],
@ -2462,7 +2993,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -2472,33 +3003,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "prod",
-              "value": "prod"
-            },
-            "datasource": "$datasource",
-            "hide": 2,
-            "includeAll": false,
-            "label": "cluster",
-            "multi": false,
-            "name": "cluster",
-            "options": [
-
-            ],
-            "query": "label_values(node_cpu_seconds_total, cluster)",
-            "refresh": 1,
-            "regex": "",
-            "sort": 2,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -2514,7 +3018,7 @@ data:
            "options": [

            ],
-            "query": "label_values(node_cpu_seconds_total, cluster)",
+            "query": "label_values(up{job=\"kubernetes-cadvisor\"}, cluster)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -2557,7 +3061,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Compute Resources / Cluster",
      "uid": "efa86fd1d0c121a26444b636a3f509a8",
      "version": 0
@ -2593,11 +3097,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 1,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2621,7 +3128,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"})",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -2634,7 +3141,7 @@ data:
              "title": "CPU Utilisation (from requests)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -2677,11 +3184,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 2,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2705,7 +3215,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"})",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) / sum(kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -2718,7 +3228,7 @@ data:
              "title": "CPU Utilisation (from limits)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -2761,11 +3271,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 3,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2789,7 +3302,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"})",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) / sum(kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -2799,10 +3312,10 @@ data:
              "thresholds": "70,80",
              "timeFrom": null,
              "timeShift": null,
-              "title": "Memory Utilization (from requests)",
+              "title": "Memory Utilisation (from requests)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -2845,11 +3358,14 @@ data:
              "fill": 1,
              "format": "percentunit",
              "id": 4,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2873,7 +3389,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"})",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) / sum(kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"})",
                  "format": "time_series",
                  "instant": true,
                  "intervalFactor": 2,
@ -2886,7 +3402,7 @@ data:
              "title": "Memory Utilisation (from limits)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "singlestat",
@ -2940,11 +3456,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 5,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2965,8 +3484,9 @@ data:
                  "color": "#F2495C",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                },
@ -2975,8 +3495,9 @@ data:
                  "color": "#FF9830",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                }
@ -2987,7 +3508,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3019,7 +3540,7 @@ data:
              "title": "CPU Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3073,11 +3594,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 6,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3115,6 +3639,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -3133,6 +3658,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -3151,6 +3677,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -3169,6 +3696,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -3187,6 +3715,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -3205,8 +3734,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                  "pattern": "pod",
                  "thresholds": [

@ -3232,7 +3762,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3241,7 +3771,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3250,7 +3780,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3259,7 +3789,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3268,7 +3798,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3285,7 +3815,7 @@ data:
              "title": "CPU Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -3340,11 +3870,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 7,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3365,8 +3898,9 @@ data:
                  "color": "#F2495C",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                },
@ -3375,8 +3909,9 @@ data:
                  "color": "#FF9830",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                }
@ -3387,7 +3922,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3419,7 +3954,7 @@ data:
              "title": "Memory Usage (w/o cache)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3473,11 +4008,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 8,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3515,6 +4053,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -3533,6 +4072,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -3551,6 +4091,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -3569,6 +4110,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -3587,6 +4129,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -3605,6 +4148,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -3623,6 +4167,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #G",
@ -3641,6 +4186,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #H",
@ -3659,8 +4205,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                  "pattern": "pod",
                  "thresholds": [

@ -3686,7 +4233,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3695,7 +4242,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3704,7 +4251,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3713,7 +4260,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3722,7 +4269,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\"}) by (pod)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\", image!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3731,7 +4278,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3740,7 +4287,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_cache{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3749,7 +4296,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
+                  "expr": "sum(container_memory_swap{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -3766,7 +4313,7 @@ data:
              "title": "Memory Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -3821,11 +4368,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 9,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3863,6 +4413,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -3881,6 +4432,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -3899,6 +4451,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -3917,6 +4470,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -3935,6 +4489,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -3953,6 +4508,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -3971,8 +4527,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                  "pattern": "pod",
                  "thresholds": [

@ -3998,7 +4555,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4007,7 +4564,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4016,7 +4573,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4025,7 +4582,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4034,7 +4591,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4043,7 +4600,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4060,7 +4617,7 @@ data:
              "title": "Current Network Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -4098,7 +4655,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
          "titleSize": "h6"
        },
        {
@ -4115,11 +4672,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 10,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4138,12 +4698,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4159,7 +4719,7 @@ data:
              "title": "Receive Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4190,19 +4750,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -4213,11 +4761,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 11,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4236,12 +4787,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4257,7 +4808,7 @@ data:
              "title": "Transmit Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4294,7 +4845,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
          "titleSize": "h6"
        },
        {
@ -4311,11 +4862,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 12,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4334,12 +4888,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4355,7 +4909,7 @@ data:
              "title": "Rate of Received Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4370,7 +4924,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -4386,19 +4940,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -4409,11 +4951,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 13,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4432,12 +4977,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4453,7 +4998,7 @@ data:
              "title": "Rate of Transmitted Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4468,7 +5013,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -4490,7 +5035,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
          "titleSize": "h6"
        },
        {
@ -4507,11 +5052,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 14,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4530,12 +5078,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4551,7 +5099,7 @@ data:
              "title": "Rate of Received Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4566,7 +5114,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -4582,19 +5130,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -4605,11 +5141,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 15,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4628,12 +5167,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4649,7 +5188,198 @@ data:
              "title": "Rate of Transmitted Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Rate of Packets Dropped",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 16,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(pod) (rate(container_fs_reads_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS(Reads+Writes)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 17,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{container!=\"\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{pod}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut(Read+Write)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4686,7 +5416,315 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Storage IO",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 18,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "sort": {
+                "col": 4,
+                "desc": true
+              },
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "IOPS(Reads)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Reads + Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Throughput(Read)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Read + Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Pod",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": true,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down to pods",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "pattern": "pod",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Current Storage IO",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution",
          "titleSize": "h6"
        }
      ],
@ -4703,7 +5741,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -4728,8 +5766,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, cluster)",
-            "refresh": 1,
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -4755,8 +5793,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "query": "label_values(kube_namespace_status_phase{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -4798,7 +5836,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Compute Resources / Namespace (Pods)",
      "uid": "85a562078cdf77779eaa1add43ccec1e",
      "version": 0
@ -4833,11 +5871,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 1,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4853,7 +5894,17 @@ data:
              "points": false,
              "renderer": "flot",
              "seriesOverrides": [
-
+                {
+                  "alias": "max capacity",
+                  "color": "#F2495C",
+                  "dashes": true,
+                  "fill": 0,
+                  "hiddenSeries": true,
+                  "hideTooltip": true,
+                  "legend": true,
+                  "linewidth": 2,
+                  "stack": false
+                }
              ],
              "spaceLength": 10,
              "span": 12,
@ -4861,7 +5912,15 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(kube_node_status_capacity{cluster=\"$cluster\", node=~\"$node\", resource=\"cpu\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "max capacity",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -4877,7 +5936,7 @@ data:
              "title": "CPU Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4931,11 +5990,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 2,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4973,6 +6035,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -4991,6 +6054,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -5009,6 +6073,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -5027,6 +6092,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -5045,6 +6111,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -5063,6 +6130,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "pod",
@ -5090,7 +6158,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5099,7 +6167,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5108,7 +6176,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=\"$node\"}) by (pod) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5117,7 +6185,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5126,7 +6194,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", node=\"$node\"}) by (pod) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", node=~\"$node\"}) by (pod) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5143,7 +6211,7 @@ data:
              "title": "CPU Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -5198,11 +6266,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 3,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5218,7 +6289,17 @@ data:
              "points": false,
              "renderer": "flot",
              "seriesOverrides": [
-
+                {
+                  "alias": "max capacity",
+                  "color": "#F2495C",
+                  "dashes": true,
+                  "fill": 0,
+                  "hiddenSeries": true,
+                  "hideTooltip": true,
+                  "legend": true,
+                  "linewidth": 2,
+                  "stack": false
+                }
              ],
              "spaceLength": 10,
              "span": 12,
@ -5226,7 +6307,15 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=\"$node\", container!=\"\"}) by (pod)",
+                  "expr": "sum(kube_node_status_capacity{cluster=\"$cluster\", node=~\"$node\", resource=\"memory\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "max capacity",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\", container!=\"\"}) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -5242,7 +6331,7 @@ data:
              "title": "Memory Usage (w/o cache)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5296,11 +6385,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 4,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5338,6 +6430,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -5356,6 +6449,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -5374,6 +6468,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -5392,6 +6487,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -5410,6 +6506,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -5428,6 +6525,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -5446,6 +6544,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #G",
@ -5464,6 +6563,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #H",
@ -5482,6 +6582,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "pod",
@ -5509,7 +6610,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=\"$node\",container!=\"\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5518,7 +6619,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5527,7 +6628,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=\"$node\",container!=\"\"}) by (pod) / sum(kube_pod_container_resource_requests_memory_bytes{node=\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5536,7 +6637,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", node=\"$node\"}) by (pod)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5545,7 +6646,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=\"$node\",container!=\"\"}) by (pod) / sum(kube_pod_container_resource_limits_memory_bytes{node=\"$node\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_working_set_bytes{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", node=~\"$node\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5554,7 +6655,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_rss{cluster=\"$cluster\", node=\"$node\",container!=\"\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_rss{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5563,7 +6664,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_cache{cluster=\"$cluster\", node=\"$node\",container!=\"\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_cache{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5572,7 +6673,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_memory_swap{cluster=\"$cluster\", node=\"$node\",container!=\"\"}) by (pod)",
+                  "expr": "sum(node_namespace_pod_container:container_memory_swap{cluster=\"$cluster\", node=~\"$node\",container!=\"\"}) by (pod)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -5589,7 +6690,7 @@ data:
              "title": "Memory Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -5644,7 +6745,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -5669,8 +6770,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, cluster)",
-            "refresh": 1,
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -5691,13 +6792,13 @@ data:
            "hide": 0,
            "includeAll": false,
            "label": null,
-            "multi": false,
+            "multi": true,
            "name": "node",
            "options": [

            ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, node)",
-            "refresh": 1,
+            "query": "label_values(kube_node_info{cluster=\"$cluster\"}, node)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -5739,7 +6840,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Compute Resources / Node (Pods)",
      "uid": "200ac8fdbfbb74b39aff88118e4d1c2c",
      "version": 0
--- a/addons/grafana/dashboards-k8s-resources-2.yaml
+++ b/addons/grafana/dashboards-k8s-resources-2.yaml
@ -30,11 +30,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 1,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -75,7 +78,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", cluster=\"$cluster\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace=\"$namespace\", pod=\"$pod\", cluster=\"$cluster\"}) by (container)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{container}}",
@ -83,7 +86,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"cpu\"}\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "requests",
@ -91,7 +94,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"cpu\"}\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "limits",
@ -107,7 +110,7 @@ data:
              "title": "CPU Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -161,11 +164,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 2,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": true,
                "max": true,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -189,7 +195,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(increase(container_cpu_cfs_throttled_periods_total{namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", cluster=\"$cluster\"}[5m])) by (container) /sum(increase(container_cpu_cfs_periods_total{namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", cluster=\"$cluster\"}[5m])) by (container)",
+                  "expr": "sum(increase(container_cpu_cfs_throttled_periods_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", cluster=\"$cluster\"}[$__rate_interval])) by (container) /sum(increase(container_cpu_cfs_periods_total{job=\"kubernetes-cadvisor\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", cluster=\"$cluster\"}[$__rate_interval])) by (container)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{container}}",
@ -203,7 +209,7 @@ data:
                  "fill": true,
                  "line": true,
                  "op": "gt",
-                  "value": 1,
+                  "value": 0.80000000000000004,
                  "yaxis": "left"
                }
              ],
@ -212,7 +218,7 @@ data:
              "title": "CPU Throttling",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -266,11 +272,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 3,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -308,6 +317,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -326,6 +336,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -344,6 +355,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -362,6 +374,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -380,6 +393,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -398,6 +412,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "container",
@ -425,7 +440,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -434,7 +449,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -443,7 +458,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -452,7 +467,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -461,7 +476,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -478,7 +493,7 @@ data:
              "title": "CPU Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -533,11 +548,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 4,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -559,7 +577,7 @@ data:
                  "dashes": true,
                  "fill": 0,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                },
@ -569,7 +587,7 @@ data:
                  "dashes": true,
                  "fill": 0,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                }
@ -580,7 +598,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", container!=\"\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{container}}",
@ -588,7 +606,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"memory\"}\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "requests",
@ -596,7 +614,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"})\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"memory\"}\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "limits",
@ -609,10 +627,10 @@ data:
              ],
              "timeFrom": null,
              "timeShift": null,
-              "title": "Memory Usage",
+              "title": "Memory Usage (WSS)",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -666,11 +684,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 5,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -700,7 +721,7 @@ data:
                  "type": "hidden"
                },
                {
-                  "alias": "Memory Usage",
+                  "alias": "Memory Usage (WSS)",
                  "colorMode": null,
                  "colors": [

@ -708,6 +729,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -726,6 +748,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -744,6 +767,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -762,6 +786,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -780,6 +805,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -798,6 +824,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -816,6 +843,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #G",
@ -834,6 +862,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #H",
@ -852,6 +881,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "container",
@ -879,7 +909,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"POD\", container!=\"\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -888,7 +918,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -897,7 +927,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", image!=\"\"}) by (container) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_requests{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -906,7 +936,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\"}) by (container)",
+                  "expr": "sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -915,7 +945,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\"}) by (container) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
+                  "expr": "sum(container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container) / sum(cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -924,7 +954,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
+                  "expr": "sum(container_memory_rss{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -933,7 +963,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_cache{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
+                  "expr": "sum(container_memory_cache{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -942,7 +972,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(container_memory_swap{cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
+                  "expr": "sum(container_memory_swap{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container != \"\", container != \"POD\"}) by (container)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -959,7 +989,7 @@ data:
              "title": "Memory Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -1014,11 +1044,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 6,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1037,12 +1070,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1058,7 +1091,7 @@ data:
              "title": "Receive Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1089,19 +1122,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -1112,11 +1133,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 7,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1135,12 +1159,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1156,7 +1180,7 @@ data:
              "title": "Transmit Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1193,7 +1217,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
          "titleSize": "h6"
        },
        {
@ -1210,11 +1234,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 8,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1233,12 +1260,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1254,7 +1281,7 @@ data:
              "title": "Rate of Received Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1269,7 +1296,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -1285,19 +1312,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -1308,11 +1323,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 9,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1331,12 +1349,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1352,7 +1370,7 @@ data:
              "title": "Rate of Transmitted Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1367,7 +1385,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -1389,7 +1407,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
          "titleSize": "h6"
        },
        {
@ -1406,11 +1424,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 10,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1429,12 +1450,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1450,7 +1471,7 @@ data:
              "title": "Rate of Received Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1465,7 +1486,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -1481,19 +1502,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -1504,11 +1513,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 11,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1527,12 +1539,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__interval])) by (pod)",
+                  "expr": "sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])) by (pod)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1548,7 +1560,214 @@ data:
              "title": "Rate of Transmitted Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "pps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Rate of Packets Dropped",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 12,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(pod) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Reads",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "ceil(sum by(pod) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\",namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Writes",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 13,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(pod) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Reads",
+                  "legendLink": null,
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(pod) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "Writes",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1585,7 +1804,506 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Storage IO - Distribution(Pod - Read & Writes)",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "decimals": -1,
+              "fill": 10,
+              "id": 14,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "ceil(sum by(container) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval])))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "IOPS(Reads+Writes)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 10,
+              "id": 15,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 0,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 6,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{container}}",
+                  "legendLink": null,
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "ThroughPut(Read+Write)",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "Bps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution(Containers)",
+          "titleSize": "h6"
+        },
+        {
+          "collapse": false,
+          "height": "250px",
+          "panels": [
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "fill": 1,
+              "id": 16,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null as zero",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "seriesOverrides": [
+
+              ],
+              "sort": {
+                "col": 4,
+                "desc": true
+              },
+              "spaceLength": 10,
+              "span": 12,
+              "stack": false,
+              "steppedLine": false,
+              "styles": [
+                {
+                  "alias": "Time",
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "pattern": "Time",
+                  "type": "hidden"
+                },
+                {
+                  "alias": "IOPS(Reads)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #A",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #B",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "IOPS(Reads + Writes)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": -1,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #C",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "Throughput(Read)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #D",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #E",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Throughput(Read + Write)",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "Value #F",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "Bps"
+                },
+                {
+                  "alias": "Container",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "link": false,
+                  "linkTargetBlank": false,
+                  "linkTooltip": "Drill down",
+                  "linkUrl": "",
+                  "pattern": "container",
+                  "thresholds": [
+
+                  ],
+                  "type": "number",
+                  "unit": "short"
+                },
+                {
+                  "alias": "",
+                  "colorMode": null,
+                  "colors": [
+
+                  ],
+                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
+                  "decimals": 2,
+                  "pattern": "/.*/",
+                  "thresholds": [
+
+                  ],
+                  "type": "string",
+                  "unit": "short"
+                }
+              ],
+              "targets": [
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_writes_total{job=\"kubernetes-cadvisor\",device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "B",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "C",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "D",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "E",
+                  "step": 10
+                },
+                {
+                  "expr": "sum by(container) (rate(container_fs_reads_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]) + rate(container_fs_writes_bytes_total{job=\"kubernetes-cadvisor\", device=~\"mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\", container!=\"\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[$__rate_interval]))",
+                  "format": "table",
+                  "instant": true,
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "F",
+                  "step": 10
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Current Storage IO",
+              "tooltip": {
+                "shared": false,
+                "sort": 2,
+                "value_type": "individual"
+              },
+              "transform": "table",
+              "type": "table",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "short",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": false
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": true,
+          "title": "Storage IO - Distribution",
          "titleSize": "h6"
        }
      ],
@ -1602,7 +2320,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -1627,8 +2345,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, cluster)",
-            "refresh": 1,
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -1654,8 +2372,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "query": "label_values(kube_namespace_status_phase{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -1681,7 +2399,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=\"$namespace\"}, pod)",
+            "query": "label_values(kube_pod_info{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\"}, pod)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -1724,7 +2442,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Compute Resources / Pod",
      "uid": "6581e46e4e5c7ba40a07646395ef7b23",
      "version": 0
@ -1759,11 +2477,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 1,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1787,7 +2508,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -1803,7 +2524,7 @@ data:
              "title": "CPU Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -1857,11 +2578,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 2,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -1899,6 +2623,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -1917,6 +2642,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -1935,6 +2661,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -1953,6 +2680,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -1971,6 +2699,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -1989,8 +2718,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                  "pattern": "pod",
                  "thresholds": [

@ -2016,7 +2746,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2025,7 +2755,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2034,7 +2764,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2043,7 +2773,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2052,7 +2782,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2069,7 +2799,7 @@ data:
              "title": "CPU Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -2124,11 +2854,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 3,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2152,7 +2885,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -2168,7 +2901,7 @@ data:
              "title": "Memory Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2222,11 +2955,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 4,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2264,6 +3000,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -2282,6 +3019,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -2300,6 +3038,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -2318,6 +3057,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -2336,6 +3076,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -2354,8 +3095,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                  "pattern": "pod",
                  "thresholds": [

@ -2381,7 +3123,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2390,7 +3132,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2399,7 +3141,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2408,7 +3150,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2417,7 +3159,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n/sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\", workload_type=\"$type\"}\n) by (pod)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2434,7 +3176,7 @@ data:
              "title": "Memory Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -2489,11 +3231,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 5,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2531,6 +3276,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -2549,6 +3295,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -2567,6 +3314,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -2585,6 +3333,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -2603,6 +3352,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -2621,6 +3371,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -2639,8 +3390,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
+                  "linkUrl": "/d/6581e46e4e5c7ba40a07646395ef7b23/k8s-resources-pod?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-pod=$__cell",
                  "pattern": "pod",
                  "thresholds": [

@ -2666,7 +3418,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2675,7 +3427,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2684,7 +3436,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2693,7 +3445,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2702,7 +3454,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2711,7 +3463,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -2728,7 +3480,7 @@ data:
              "title": "Current Network Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -2766,7 +3518,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
          "titleSize": "h6"
        },
        {
@ -2783,11 +3535,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 6,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2806,12 +3561,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -2827,7 +3582,7 @@ data:
              "title": "Receive Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2858,19 +3613,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -2881,11 +3624,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 7,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -2904,12 +3650,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -2925,7 +3671,7 @@ data:
              "title": "Transmit Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2962,7 +3708,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
          "titleSize": "h6"
        },
        {
@ -2979,11 +3725,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 8,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3002,12 +3751,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3023,7 +3772,7 @@ data:
              "title": "Average Container Bandwidth by Pod: Received",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3054,19 +3803,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -3077,11 +3814,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 9,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3100,12 +3840,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3121,7 +3861,7 @@ data:
              "title": "Average Container Bandwidth by Pod: Transmitted",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3158,7 +3898,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Average Container Bandwidth by Pod",
          "titleSize": "h6"
        },
        {
@ -3175,11 +3915,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 10,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3198,12 +3941,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3219,7 +3962,7 @@ data:
              "title": "Rate of Received Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3234,7 +3977,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -3250,19 +3993,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -3273,11 +4004,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 11,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3296,12 +4030,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3317,7 +4051,7 @@ data:
              "title": "Rate of Transmitted Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3332,7 +4066,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -3354,7 +4088,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
          "titleSize": "h6"
        },
        {
@ -3371,11 +4105,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 12,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3394,12 +4131,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3415,7 +4152,7 @@ data:
              "title": "Rate of Received Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3430,7 +4167,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -3446,19 +4183,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -3469,11 +4194,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 13,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3492,12 +4220,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\"$workload\", workload_type=\"$type\"}) by (pod))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{pod}}",
@ -3513,7 +4241,7 @@ data:
              "title": "Rate of Transmitted Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3528,7 +4256,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -3550,7 +4278,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets Dropped",
          "titleSize": "h6"
        }
      ],
@ -3567,7 +4295,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -3592,8 +4320,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, cluster)",
-            "refresh": 1,
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -3619,35 +4347,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
-            "regex": "",
-            "sort": 1,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
-          {
-            "allValue": null,
-            "current": {
-              "text": "",
-              "value": ""
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "workload",
-            "options": [
-
-            ],
-            "query": "label_values(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\"}, workload)",
-            "refresh": 1,
+            "query": "label_values(kube_namespace_status_phase{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -3673,8 +4374,35 @@ data:
            "options": [

            ],
-            "query": "label_values(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload=\"$workload\"}, workload_type)",
-            "refresh": 1,
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+              "text": "",
+              "value": ""
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "workload",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}, workload)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -3716,7 +4444,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Compute Resources / Workload",
      "uid": "a164a7f0339f99e89cea5cb47e9be617",
      "version": 0
@ -3751,11 +4479,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 1,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3776,8 +4507,9 @@ data:
                  "color": "#F2495C",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                },
@ -3786,8 +4518,9 @@ data:
                  "color": "#FF9830",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                }
@ -3798,7 +4531,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}} - {{workload_type}}",
@ -3830,7 +4563,7 @@ data:
              "title": "CPU Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -3884,11 +4617,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 2,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -3926,6 +4662,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 0,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -3944,6 +4681,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -3962,6 +4700,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -3980,6 +4719,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -3998,6 +4738,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -4016,6 +4757,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -4034,8 +4776,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
                  "pattern": "workload",
                  "thresholds": [

@ -4052,6 +4795,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "workload_type",
@ -4079,7 +4823,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload, workload_type)",
+                  "expr": "count(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload, workload_type)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4088,7 +4832,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4097,7 +4841,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4106,7 +4850,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4115,7 +4859,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4124,7 +4868,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits_cpu_cores{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"cpu\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4141,7 +4885,7 @@ data:
              "title": "CPU Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -4196,11 +4940,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 3,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4221,8 +4968,9 @@ data:
                  "color": "#F2495C",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                },
@ -4231,8 +4979,9 @@ data:
                  "color": "#FF9830",
                  "dashes": true,
                  "fill": 0,
+                  "hiddenSeries": true,
                  "hideTooltip": true,
-                  "legend": false,
+                  "legend": true,
                  "linewidth": 2,
                  "stack": false
                }
@ -4243,7 +4992,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}} - {{workload_type}}",
@ -4275,7 +5024,7 @@ data:
              "title": "Memory Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -4329,11 +5078,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 4,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4371,6 +5123,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 0,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -4389,6 +5142,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -4407,6 +5161,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -4425,6 +5180,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -4443,6 +5199,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -4461,6 +5218,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -4479,8 +5237,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
-                  "linkUrl": "./d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$__cell_2",
                  "pattern": "workload",
                  "thresholds": [

@ -4497,6 +5256,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "workload_type",
@ -4524,7 +5284,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "count(mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload, workload_type)",
+                  "expr": "count(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload, workload_type)",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4533,7 +5293,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4542,7 +5302,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4551,7 +5311,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4560,7 +5320,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4569,7 +5329,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "sum(\n    container_memory_working_set_bytes{cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits_memory_bytes{cluster=\"$cluster\", namespace=\"$namespace\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
+                  "expr": "sum(\n    container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", container!=\"\", image!=\"\"}\n  * on(namespace,pod)\n    group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n/sum(\n  kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", resource=\"memory\"}\n* on(namespace,pod)\n  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}\n) by (workload, workload_type)\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4586,7 +5346,7 @@ data:
              "title": "Memory Quota",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -4641,11 +5401,14 @@ data:
              "datasource": "$datasource",
              "fill": 1,
              "id": 5,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4683,6 +5446,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -4701,6 +5465,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -4719,6 +5484,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #C",
@ -4737,6 +5503,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #D",
@ -4755,6 +5522,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #E",
@ -4773,6 +5541,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #F",
@ -4791,8 +5560,9 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": true,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down to pods",
-                  "linkUrl": "./d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$type",
+                  "linkUrl": "/d/a164a7f0339f99e89cea5cb47e9be617/k8s-resources-workload?var-datasource=$datasource&var-cluster=$cluster&var-namespace=$namespace&var-workload=$__cell&var-type=$type",
                  "pattern": "workload",
                  "thresholds": [

@ -4809,6 +5579,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "workload_type",
@ -4836,7 +5607,7 @@ data:
              ],
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4845,7 +5616,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4854,7 +5625,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4863,7 +5634,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4872,7 +5643,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4881,7 +5652,7 @@ data:
                  "step": 10
                },
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "table",
                  "instant": true,
                  "intervalFactor": 2,
@ -4898,7 +5669,7 @@ data:
              "title": "Current Network Usage",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -4936,7 +5707,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Current Network Usage",
          "titleSize": "h6"
        },
        {
@ -4953,11 +5724,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 6,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -4976,12 +5750,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -4997,7 +5771,7 @@ data:
              "title": "Receive Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5028,19 +5802,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -5051,11 +5813,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 7,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5074,12 +5839,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5095,7 +5860,7 @@ data:
              "title": "Transmit Bandwidth",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5132,7 +5897,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Bandwidth",
          "titleSize": "h6"
        },
        {
@ -5149,11 +5914,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 8,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5172,12 +5940,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(avg(irate(container_network_receive_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5193,7 +5961,7 @@ data:
              "title": "Average Container Bandwidth by Workload: Received",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5224,19 +5992,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -5247,11 +6003,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 9,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5270,12 +6029,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(avg(irate(container_network_transmit_bytes_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(avg(irate(container_network_transmit_bytes_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5291,7 +6050,7 @@ data:
              "title": "Average Container Bandwidth by Workload: Transmitted",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5328,7 +6087,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Average Container Bandwidth by Workload",
          "titleSize": "h6"
        },
        {
@ -5345,11 +6104,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 10,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5368,12 +6130,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5389,7 +6151,7 @@ data:
              "title": "Rate of Received Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5404,7 +6166,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -5420,19 +6182,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -5443,11 +6193,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 11,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5466,12 +6219,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5487,7 +6240,7 @@ data:
              "title": "Rate of Transmitted Packets",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5502,7 +6255,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -5524,7 +6277,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets",
          "titleSize": "h6"
        },
        {
@ -5541,11 +6294,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 12,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5564,12 +6320,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_receive_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5585,7 +6341,7 @@ data:
              "title": "Rate of Received Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5600,7 +6356,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -5616,19 +6372,7 @@ data:
                  "show": false
                }
              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": true,
-          "title": "Network",
-          "titleSize": "h6"
-        },
-        {
-          "collapse": false,
-          "height": "250px",
-          "panels": [
+            },
            {
              "aliasColors": {

@ -5639,11 +6383,14 @@ data:
              "datasource": "$datasource",
              "fill": 10,
              "id": 13,
+              "interval": "1m",
              "legend": {
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
+                "rightSide": true,
                "show": true,
                "total": false,
                "values": false
@ -5662,12 +6409,12 @@ data:

              ],
              "spaceLength": 10,
-              "span": 12,
+              "span": 6,
              "stack": true,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{cluster=\"$cluster\", namespace=~\"$namespace\"}[$__interval])\n* on (namespace,pod) \ngroup_left(workload,workload_type) mixin_pod_workload{cluster=\"$cluster\", namespace=~\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
+                  "expr": "(sum(irate(container_network_transmit_packets_dropped_total{job=\"kubernetes-cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\"}[$__rate_interval])\n* on (namespace,pod)\ngroup_left(workload,workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\", workload_type=\"$type\"}) by (workload))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{workload}}",
@ -5683,7 +6430,7 @@ data:
              "title": "Rate of Transmitted Packets Dropped",
              "tooltip": {
                "shared": false,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -5698,7 +6445,7 @@ data:
              },
              "yaxes": [
                {
-                  "format": "Bps",
+                  "format": "pps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -5720,7 +6467,7 @@ data:
          "repeatIteration": null,
          "repeatRowId": null,
          "showTitle": true,
-          "title": "Network",
+          "title": "Rate of Packets Dropped",
          "titleSize": "h6"
        }
      ],
@ -5737,7 +6484,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -5747,38 +6494,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "auto": false,
-            "auto_count": 30,
-            "auto_min": "10s",
-            "current": {
-              "text": "deployment",
-              "value": "deployment"
-            },
-            "datasource": "$datasource",
-            "definition": "label_values(mixin_pod_workload{namespace=~\"$namespace\", workload=~\".+\"}, workload_type)",
-            "hide": 0,
-            "includeAll": false,
-            "label": null,
-            "multi": false,
-            "name": "type",
-            "options": [
-
-            ],
-            "query": "label_values(mixin_pod_workload{namespace=~\"$namespace\", workload=~\".+\"}, workload_type)",
-            "refresh": 1,
-            "regex": "",
-            "skipUrlSync": false,
-            "sort": 0,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -5794,8 +6509,8 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info, cluster)",
-            "refresh": 1,
+            "query": "label_values(up{job=\"kube-state-metrics\"}, cluster)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -5821,13 +6536,45 @@ data:
            "options": [

            ],
-            "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
-            "refresh": 1,
+            "query": "label_values(kube_pod_info{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
            "tags": [

+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "text": "deployment",
+              "value": "deployment"
+            },
+            "datasource": "$datasource",
+            "definition": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "type",
+            "options": [
+
+            ],
+            "query": "label_values(namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload=~\".+\"}, workload_type)",
+            "refresh": 2,
+            "regex": "",
+            "skipUrlSync": false,
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
            ],
            "tagsQuery": "",
            "type": "query",
@ -5864,7 +6611,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Compute Resources / Namespace (Workloads)",
      "uid": "a87fb0d919ec0ea5f6543124e16c42a5",
      "version": 0
--- a/addons/grafana/dashboards-k8s.yaml
+++ b/addons/grafana/dashboards-k8s.yaml
@ -20,6 +20,24 @@ data:
      "id": null,
      "links": [

+      ],
+      "panels": [
+        {
+          "content": "The SLO (service level objective) and other metrics displayed on this dashboard are for informational purposes only.",
+          "datasource": null,
+          "description": "The SLO (service level objective) and other metrics displayed on this dashboard are for informational purposes only.",
+          "gridPos": {
+            "h": 2,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 2,
+          "mode": "markdown",
+          "span": 12,
+          "title": "Notice",
+          "type": "text"
+        }
      ],
      "refresh": "10s",
      "rows": [
@ -37,7 +55,9 @@ data:
                "#d44a3a"
              ],
              "datasource": "$datasource",
-              "format": "none",
+              "decimals": 3,
+              "description": "How many percent of requests (both read and write) in 30 days have been answered successfully and fast enough?",
+              "format": "percentunit",
              "gauge": {
                "maxValue": 100,
                "minValue": 0,
@ -48,8 +68,12 @@ data:
              "gridPos": {

              },
-              "id": 2,
-              "interval": null,
+              "id": 3,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
              "links": [

              ],
@ -78,7 +102,7 @@ data:
                  "to": "null"
                }
              ],
-              "span": 2,
+              "span": 4,
              "sparkline": {
                "fillColor": "rgba(31, 118, 189, 0.18)",
                "full": false,
@ -88,7 +112,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(up{job=\"apiserver\", cluster=\"$cluster\"})",
+                  "expr": "apiserver_request:availability30d{verb=\"all\", cluster=\"$cluster\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -96,7 +120,7 @@ data:
                }
              ],
              "thresholds": "",
-              "title": "Up",
+              "title": "Availability (30d) > 99.000%",
              "tooltip": {
                "shared": false
              },
@ -109,7 +133,7 @@ data:
                  "value": "null"
                }
              ],
-              "valueName": "min"
+              "valueName": "avg"
            },
            {
              "aliasColors": {
@ -119,19 +143,24 @@ data:
              "dashLength": 10,
              "dashes": false,
              "datasource": "$datasource",
-              "fill": 1,
+              "decimals": 3,
+              "description": "How much error budget is left looking at our 0.990% availability guarantees?",
+              "fill": 10,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 3,
+              "id": 4,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -150,37 +179,16 @@ data:

              ],
              "spaceLength": 10,
-              "span": 5,
+              "span": 8,
              "stack": false,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"2..\", cluster=\"$cluster\"}[5m]))",
+                  "expr": "100 * (apiserver_request:availability30d{verb=\"all\", cluster=\"$cluster\"} - 0.990000)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "2xx",
+                  "legendFormat": "errorbudget",
                  "refId": "A"
-                },
-                {
-                  "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"3..\", cluster=\"$cluster\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "3xx",
-                  "refId": "B"
-                },
-                {
-                  "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"4..\", cluster=\"$cluster\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "4xx",
-                  "refId": "C"
-                },
-                {
-                  "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"5..\", cluster=\"$cluster\"}[5m]))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "5xx",
-                  "refId": "D"
                }
              ],
              "thresholds": [
@ -188,7 +196,7 @@ data:
              ],
              "timeFrom": null,
              "timeShift": null,
-              "title": "RPC Rate",
+              "title": "ErrorBudget (30d) > 99.000%",
              "tooltip": {
                "shared": false,
                "sort": 0,
@ -206,7 +214,8 @@ data:
              },
              "yaxes": [
                {
-                  "format": "ops",
+                  "decimals": 3,
+                  "format": "percentunit",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -214,7 +223,221 @@ data:
                  "show": true
                },
                {
-                  "format": "ops",
+                  "decimals": 3,
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": false,
+          "title": "Dashboard Row",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "panels": [
+            {
+              "cacheTimeout": null,
+              "colorBackground": false,
+              "colorValue": false,
+              "colors": [
+                "#299c46",
+                "rgba(237, 129, 40, 0.89)",
+                "#d44a3a"
+              ],
+              "datasource": "$datasource",
+              "decimals": 3,
+              "description": "How many percent of read requests (LIST,GET) in 30 days have been answered successfully and fast enough?",
+              "format": "percentunit",
+              "gauge": {
+                "maxValue": 100,
+                "minValue": 0,
+                "show": false,
+                "thresholdLabels": false,
+                "thresholdMarkers": true
+              },
+              "gridPos": {
+
+              },
+              "id": 5,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
+              "links": [
+
+              ],
+              "mappingType": 1,
+              "mappingTypes": [
+                {
+                  "name": "value to text",
+                  "value": 1
+                },
+                {
+                  "name": "range to text",
+                  "value": 2
+                }
+              ],
+              "maxDataPoints": 100,
+              "nullPointMode": "connected",
+              "nullText": null,
+              "postfix": "",
+              "postfixFontSize": "50%",
+              "prefix": "",
+              "prefixFontSize": "50%",
+              "rangeMaps": [
+                {
+                  "from": "null",
+                  "text": "N/A",
+                  "to": "null"
+                }
+              ],
+              "span": 3,
+              "sparkline": {
+                "fillColor": "rgba(31, 118, 189, 0.18)",
+                "full": false,
+                "lineColor": "rgb(31, 120, 193)",
+                "show": false
+              },
+              "tableColumn": "",
+              "targets": [
+                {
+                  "expr": "apiserver_request:availability30d{verb=\"read\", cluster=\"$cluster\"}",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "",
+              "title": "Read Availability (30d)",
+              "tooltip": {
+                "shared": false
+              },
+              "type": "singlestat",
+              "valueFontSize": "80%",
+              "valueMaps": [
+                {
+                  "op": "=",
+                  "text": "N/A",
+                  "value": "null"
+                }
+              ],
+              "valueName": "avg"
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "description": "How many read requests (LIST,GET) per second do the apiservers get by code?",
+              "fill": 10,
+              "fillGradient": 0,
+              "gridPos": {
+
+              },
+              "id": 6,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+                {
+                  "alias": "/2../i",
+                  "color": "#56A64B"
+                },
+                {
+                  "alias": "/3../i",
+                  "color": "#F2CC0C"
+                },
+                {
+                  "alias": "/4../i",
+                  "color": "#3274D9"
+                },
+                {
+                  "alias": "/5../i",
+                  "color": "#E02F44"
+                }
+              ],
+              "spaceLength": 10,
+              "span": 3,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by (code) (code_resource:apiserver_request_total:rate5m{verb=\"read\", cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{ code }}",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Read SLI - Requests",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "reqps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "reqps",
                  "label": null,
                  "logBase": 1,
                  "max": null,
@ -231,21 +454,25 @@ data:
              "dashLength": 10,
              "dashes": false,
              "datasource": "$datasource",
+              "description": "How many percent of read requests (LIST,GET) per second are returned with errors (5xx)?",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 4,
+              "id": 7,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
-                "current": true,
+                "current": false,
                "max": false,
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
-                "values": true
+                "values": false
              },
              "lines": true,
              "linewidth": 1,
@ -262,15 +489,15 @@ data:

              ],
              "spaceLength": 10,
-              "span": 5,
+              "span": 3,
              "stack": false,
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", verb!=\"WATCH\", cluster=\"$cluster\"}[5m])) by (verb, le))",
+                  "expr": "sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"read\",code=~\"5..\", cluster=\"$cluster\"}) / sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"read\", cluster=\"$cluster\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{verb}}",
+                  "legendFormat": "{{ resource }}",
                  "refId": "A"
                }
              ],
@ -279,7 +506,505 @@ data:
              ],
              "timeFrom": null,
              "timeShift": null,
-              "title": "Request duration 99th quantile",
+              "title": "Read SLI - Errors",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "description": "How many seconds is the 99th percentile for reading (LIST|GET) a given resource?",
+              "fill": 1,
+              "fillGradient": 0,
+              "gridPos": {
+
+              },
+              "id": 8,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 3,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{verb=\"read\", cluster=\"$cluster\"}",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{ resource }}",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Read SLI - Duration",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "s",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "s",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            }
+          ],
+          "repeat": null,
+          "repeatIteration": null,
+          "repeatRowId": null,
+          "showTitle": false,
+          "title": "Dashboard Row",
+          "titleSize": "h6",
+          "type": "row"
+        },
+        {
+          "collapse": false,
+          "collapsed": false,
+          "panels": [
+            {
+              "cacheTimeout": null,
+              "colorBackground": false,
+              "colorValue": false,
+              "colors": [
+                "#299c46",
+                "rgba(237, 129, 40, 0.89)",
+                "#d44a3a"
+              ],
+              "datasource": "$datasource",
+              "decimals": 3,
+              "description": "How many percent of write requests (POST|PUT|PATCH|DELETE) in 30 days have been answered successfully and fast enough?",
+              "format": "percentunit",
+              "gauge": {
+                "maxValue": 100,
+                "minValue": 0,
+                "show": false,
+                "thresholdLabels": false,
+                "thresholdMarkers": true
+              },
+              "gridPos": {
+
+              },
+              "id": 9,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
+              "links": [
+
+              ],
+              "mappingType": 1,
+              "mappingTypes": [
+                {
+                  "name": "value to text",
+                  "value": 1
+                },
+                {
+                  "name": "range to text",
+                  "value": 2
+                }
+              ],
+              "maxDataPoints": 100,
+              "nullPointMode": "connected",
+              "nullText": null,
+              "postfix": "",
+              "postfixFontSize": "50%",
+              "prefix": "",
+              "prefixFontSize": "50%",
+              "rangeMaps": [
+                {
+                  "from": "null",
+                  "text": "N/A",
+                  "to": "null"
+                }
+              ],
+              "span": 3,
+              "sparkline": {
+                "fillColor": "rgba(31, 118, 189, 0.18)",
+                "full": false,
+                "lineColor": "rgb(31, 120, 193)",
+                "show": false
+              },
+              "tableColumn": "",
+              "targets": [
+                {
+                  "expr": "apiserver_request:availability30d{verb=\"write\", cluster=\"$cluster\"}",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": "",
+              "title": "Write Availability (30d)",
+              "tooltip": {
+                "shared": false
+              },
+              "type": "singlestat",
+              "valueFontSize": "80%",
+              "valueMaps": [
+                {
+                  "op": "=",
+                  "text": "N/A",
+                  "value": "null"
+                }
+              ],
+              "valueName": "avg"
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "description": "How many write requests (POST|PUT|PATCH|DELETE) per second do the apiservers get by code?",
+              "fill": 10,
+              "fillGradient": 0,
+              "gridPos": {
+
+              },
+              "id": 10,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+                {
+                  "alias": "/2../i",
+                  "color": "#56A64B"
+                },
+                {
+                  "alias": "/3../i",
+                  "color": "#F2CC0C"
+                },
+                {
+                  "alias": "/4../i",
+                  "color": "#3274D9"
+                },
+                {
+                  "alias": "/5../i",
+                  "color": "#E02F44"
+                }
+              ],
+              "spaceLength": 10,
+              "span": 3,
+              "stack": true,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by (code) (code_resource:apiserver_request_total:rate5m{verb=\"write\", cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{ code }}",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Write SLI - Requests",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "reqps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                },
+                {
+                  "format": "reqps",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": null,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "description": "How many percent of write requests (POST|PUT|PATCH|DELETE) per second are returned with errors (5xx)?",
+              "fill": 1,
+              "fillGradient": 0,
+              "gridPos": {
+
+              },
+              "id": 11,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 3,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"write\",code=~\"5..\", cluster=\"$cluster\"}) / sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"write\", cluster=\"$cluster\"})",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{ resource }}",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Write SLI - Errors",
+              "tooltip": {
+                "shared": false,
+                "sort": 0,
+                "value_type": "individual"
+              },
+              "type": "graph",
+              "xaxis": {
+                "buckets": null,
+                "mode": "time",
+                "name": null,
+                "show": true,
+                "values": [
+
+                ]
+              },
+              "yaxes": [
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                },
+                {
+                  "format": "percentunit",
+                  "label": null,
+                  "logBase": 1,
+                  "max": null,
+                  "min": 0,
+                  "show": true
+                }
+              ]
+            },
+            {
+              "aliasColors": {
+
+              },
+              "bars": false,
+              "dashLength": 10,
+              "dashes": false,
+              "datasource": "$datasource",
+              "description": "How many seconds is the 99th percentile for writing (POST|PUT|PATCH|DELETE) a given resource?",
+              "fill": 1,
+              "fillGradient": 0,
+              "gridPos": {
+
+              },
+              "id": 12,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "avg": false,
+                "current": false,
+                "max": false,
+                "min": false,
+                "rightSide": true,
+                "show": true,
+                "sideWidth": null,
+                "total": false,
+                "values": false
+              },
+              "lines": true,
+              "linewidth": 1,
+              "links": [
+
+              ],
+              "nullPointMode": "null",
+              "percentage": false,
+              "pointradius": 5,
+              "points": false,
+              "renderer": "flot",
+              "repeat": null,
+              "seriesOverrides": [
+
+              ],
+              "spaceLength": 10,
+              "span": 3,
+              "stack": false,
+              "steppedLine": false,
+              "targets": [
+                {
+                  "expr": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{verb=\"write\", cluster=\"$cluster\"}",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "{{ resource }}",
+                  "refId": "A"
+                }
+              ],
+              "thresholds": [
+
+              ],
+              "timeFrom": null,
+              "timeShift": null,
+              "title": "Write SLI - Duration",
              "tooltip": {
                "shared": false,
                "sort": 0,
@ -336,18 +1061,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 5,
+              "id": 13,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": false,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -371,7 +1099,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(workqueue_adds_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_adds_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])) by (instance, name)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} {{name}}",
@ -427,18 +1155,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 6,
+              "id": 14,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": false,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -462,7 +1193,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(workqueue_depth{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_depth{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])) by (instance, name)",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} {{name}}",
@ -518,10 +1249,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 7,
+              "id": 15,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -530,6 +1263,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -553,7 +1287,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])) by (instance, name, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}} {{name}}",
@ -622,318 +1356,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 8,
+              "id": 16,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "etcd_helper_cache_entry_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}}",
-                  "refId": "A"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "ETCD Cache Entry Total",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "short",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 9,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "sum(rate(etcd_helper_cache_hit_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} hit",
-                  "refId": "A"
-                },
-                {
-                  "expr": "sum(rate(etcd_helper_cache_miss_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance)",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} miss",
-                  "refId": "B"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "ETCD Cache Hit/Miss Rate",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "ops",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            },
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 10,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
-                "show": true,
-                "total": false,
-                "values": false
-              },
-              "lines": true,
-              "linewidth": 1,
-              "links": [
-
-              ],
-              "nullPointMode": "null",
-              "percentage": false,
-              "pointradius": 5,
-              "points": false,
-              "renderer": "flot",
-              "repeat": null,
-              "seriesOverrides": [
-
-              ],
-              "spaceLength": 10,
-              "span": 4,
-              "stack": false,
-              "steppedLine": false,
-              "targets": [
-                {
-                  "expr": "histogram_quantile(0.99,sum(rate(etcd_request_cache_get_duration_seconds_bucket{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} get",
-                  "refId": "A"
-                },
-                {
-                  "expr": "histogram_quantile(0.99,sum(rate(etcd_request_cache_add_duration_seconds_bucket{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, le))",
-                  "format": "time_series",
-                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} miss",
-                  "refId": "B"
-                }
-              ],
-              "thresholds": [
-
-              ],
-              "timeFrom": null,
-              "timeShift": null,
-              "title": "ETCD Cache Duration 99th Quantile",
-              "tooltip": {
-                "shared": false,
-                "sort": 0,
-                "value_type": "individual"
-              },
-              "type": "graph",
-              "xaxis": {
-                "buckets": null,
-                "mode": "time",
-                "name": null,
-                "show": true,
-                "values": [
-
-                ]
-              },
-              "yaxes": [
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                },
-                {
-                  "format": "s",
-                  "label": null,
-                  "logBase": 1,
-                  "max": null,
-                  "min": 0,
-                  "show": true
-                }
-              ]
-            }
-          ],
-          "repeat": null,
-          "repeatIteration": null,
-          "repeatRowId": null,
-          "showTitle": false,
-          "title": "Dashboard Row",
-          "titleSize": "h6",
-          "type": "row"
-        },
-        {
-          "collapse": false,
-          "collapsed": false,
-          "panels": [
-            {
-              "aliasColors": {
-
-              },
-              "bars": false,
-              "dashLength": 10,
-              "dashes": false,
-              "datasource": "$datasource",
-              "fill": 1,
-              "gridPos": {
-
-              },
-              "id": 11,
-              "legend": {
-                "alignAsTable": false,
-                "avg": false,
-                "current": false,
-                "max": false,
-                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1013,18 +1450,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 12,
+              "id": 17,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1048,7 +1488,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(process_cpu_seconds_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[$__rate_interval])",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -1104,18 +1544,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
-              "id": 13,
+              "id": 18,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1209,7 +1652,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -1222,20 +1665,19 @@ data:
          {
            "allValue": null,
            "current": {
-              "text": "prod",
-              "value": "prod"
+
            },
            "datasource": "$datasource",
            "hide": 2,
            "includeAll": false,
-            "label": null,
+            "label": "cluster",
            "multi": false,
            "name": "cluster",
            "options": [

            ],
-            "query": "label_values(apiserver_request_total, cluster)",
-            "refresh": 1,
+            "query": "label_values(up{job=\"apiserver\"}, cluster)",
+            "refresh": 2,
            "regex": "",
            "sort": 1,
            "tagValuesQuery": "",
@ -1260,7 +1702,7 @@ data:
            "options": [

            ],
-            "query": "label_values(apiserver_request_total{job=\"apiserver\", cluster=\"$cluster\"}, instance)",
+            "query": "label_values(up{job=\"apiserver\", cluster=\"$cluster\"}, instance)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -1303,7 +1745,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / API server",
      "uid": "09ec8aa1e996d6ffcd6817bbaff4db1b",
      "version": 0
@ -1357,7 +1799,11 @@ data:

              },
              "id": 2,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
              "links": [

              ],
@ -1396,7 +1842,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(up{job=\"kube-controller-manager\"})",
+                  "expr": "sum(up{cluster=\"$cluster\", job=\"kube-controller-manager\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -1428,10 +1874,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 3,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -1440,6 +1888,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -1463,10 +1912,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(workqueue_adds_total{job=\"kube-controller-manager\", instance=~\"$instance\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_adds_total{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, name)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{name}}",
+                  "legendFormat": "{{cluster}} {{instance}} {{name}}",
                  "refId": "A"
                }
              ],
@ -1532,10 +1981,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 4,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -1544,6 +1995,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -1567,10 +2019,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(workqueue_depth{job=\"kube-controller-manager\", instance=~\"$instance\"}[5m])) by (instance, name)",
+                  "expr": "sum(rate(workqueue_depth{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, name)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{name}}",
+                  "legendFormat": "{{cluster}} {{instance}} {{name}}",
                  "refId": "A"
                }
              ],
@ -1636,10 +2088,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 5,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -1648,6 +2102,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -1671,10 +2126,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"kube-controller-manager\", instance=~\"$instance\"}[5m])) by (instance, name, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, name, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} {{name}}",
+                  "legendFormat": "{{cluster}} {{instance}} {{name}}",
                  "refId": "A"
                }
              ],
@ -1740,18 +2195,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 6,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1775,28 +2233,28 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "2xx",
                  "refId": "A"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "3xx",
                  "refId": "B"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "4xx",
                  "refId": "C"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-controller-manager\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "5xx",
@ -1852,18 +2310,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 7,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1887,7 +2348,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"POST\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"POST\"}[$__rate_interval])) by (verb, url, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{verb}} {{url}}",
@ -1956,10 +2417,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 8,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -1968,6 +2431,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -1991,7 +2455,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"GET\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-controller-manager\", instance=~\"$instance\", verb=\"GET\"}[$__rate_interval])) by (verb, url, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{verb}} {{url}}",
@ -2060,18 +2524,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 9,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -2095,7 +2562,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "process_resident_memory_bytes{job=\"kube-controller-manager\",instance=~\"$instance\"}",
+                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\", job=\"kube-controller-manager\",instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -2151,18 +2618,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 10,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -2186,7 +2656,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(process_cpu_seconds_total{job=\"kube-controller-manager\",instance=~\"$instance\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", job=\"kube-controller-manager\",instance=~\"$instance\"}[$__rate_interval])",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -2242,18 +2712,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 11,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -2277,7 +2750,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "go_goroutines{job=\"kube-controller-manager\",instance=~\"$instance\"}",
+                  "expr": "go_goroutines{cluster=\"$cluster\", job=\"kube-controller-manager\",instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -2347,7 +2820,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -2361,6 +2834,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kube-controller-manager\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
@ -2371,7 +2870,7 @@ data:
            "options": [

            ],
-            "query": "label_values(process_cpu_seconds_total{job=\"kube-controller-manager\"}, instance)",
+            "query": "label_values(up{cluster=\"$cluster\", job=\"kube-controller-manager\"}, instance)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -2414,7 +2913,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Controller Manager",
      "uid": "72e0e05bef5099e5f049b05fdc429ed4",
      "version": 0
@ -2455,18 +2954,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 2,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": true,
                "current": true,
                "max": true,
                "min": true,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -2490,14 +2992,14 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(\n  sum without(instance, node) (kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  sum without(instance, node) (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n",
+                  "expr": "(\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n  -\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n)\n",
                  "format": "time_series",
                  "intervalFactor": 1,
                  "legendFormat": "Used Space",
                  "refId": "A"
                },
                {
-                  "expr": "sum without(instance, node) (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n",
+                  "expr": "sum without(instance, node) (topk(1, (kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n",
                  "format": "time_series",
                  "intervalFactor": 1,
                  "legendFormat": "Free Space",
@ -2566,7 +3068,11 @@ data:

              },
              "id": 3,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
              "links": [

              ],
@ -2605,7 +3111,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "(\n  kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n  -\n  kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n)\n/\nkubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n* 100\n",
+                  "expr": "max without(instance,node) (\n(\n  topk(1, kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  topk(1, kubelet_volume_stats_available_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n/\ntopk(1, kubelet_volume_stats_capacity_bytes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n* 100)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -2650,18 +3156,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 4,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": true,
                "current": true,
                "max": true,
                "min": true,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -2685,14 +3194,14 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum without(instance, node) (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n",
+                  "expr": "sum without(instance, node) (topk(1, (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n",
                  "format": "time_series",
                  "intervalFactor": 1,
                  "legendFormat": "Used inodes",
                  "refId": "A"
                },
                {
-                  "expr": "(\n  sum without(instance, node) (kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n  -\n  sum without(instance, node) (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n)\n",
+                  "expr": "(\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n  -\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})))\n)\n",
                  "format": "time_series",
                  "intervalFactor": 1,
                  "legendFormat": " Free inodes",
@ -2761,7 +3270,11 @@ data:

              },
              "id": 5,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
              "links": [

              ],
@ -2800,7 +3313,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n/\nkubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"}\n* 100\n",
+                  "expr": "max without(instance,node) (\ntopk(1, kubelet_volume_stats_inodes_used{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n/\ntopk(1, kubelet_volume_stats_inodes{cluster=\"$cluster\", job=\"kubelet\", namespace=\"$namespace\", persistentvolumeclaim=\"$volume\"})\n* 100)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -2846,7 +3359,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -2870,7 +3383,7 @@ data:
            "options": [

            ],
-            "query": "label_values(kubelet_volume_stats_capacity_bytes, cluster)",
+            "query": "label_values(kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}, cluster)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -2965,7 +3478,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Persistent Volumes",
      "uid": "919b92a8e8041bd567af9edab12c840c",
      "version": 0
@ -3019,7 +3532,11 @@ data:

              },
              "id": 2,
-              "interval": null,
+              "interval": "1m",
+              "legend": {
+                "alignAsTable": true,
+                "rightSide": true
+              },
              "links": [

              ],
@ -3058,7 +3575,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(up{job=\"kube-scheduler\"})",
+                  "expr": "sum(up{cluster=\"$cluster\", job=\"kube-scheduler\"})",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -3090,10 +3607,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 3,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -3102,6 +3621,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -3125,31 +3645,31 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(scheduler_e2e_scheduling_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_e2e_scheduling_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} e2e",
+                  "legendFormat": "{{cluster}} {{instance}} e2e",
                  "refId": "A"
                },
                {
-                  "expr": "sum(rate(scheduler_binding_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_binding_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} binding",
+                  "legendFormat": "{{cluster}} {{instance}} binding",
                  "refId": "B"
                },
                {
-                  "expr": "sum(rate(scheduler_scheduling_algorithm_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_scheduling_algorithm_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} scheduling algorithm",
+                  "legendFormat": "{{cluster}} {{instance}} scheduling algorithm",
                  "refId": "C"
                },
                {
-                  "expr": "sum(rate(scheduler_volume_scheduling_duration_seconds_count{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])) by (instance)",
+                  "expr": "sum(rate(scheduler_volume_scheduling_duration_seconds_count{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance)",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} volume",
+                  "legendFormat": "{{cluster}} {{instance}} volume",
                  "refId": "D"
                }
              ],
@ -3202,10 +3722,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 4,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -3214,6 +3736,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -3237,31 +3760,31 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_e2e_scheduling_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} e2e",
+                  "legendFormat": "{{cluster}} {{instance}} e2e",
                  "refId": "A"
                },
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_binding_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_binding_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} binding",
+                  "legendFormat": "{{cluster}} {{instance}} binding",
                  "refId": "B"
                },
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_scheduling_algorithm_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} scheduling algorithm",
+                  "legendFormat": "{{cluster}} {{instance}} scheduling algorithm",
                  "refId": "C"
                },
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_volume_scheduling_duration_seconds_bucket{job=\"kube-scheduler\",instance=~\"$instance\"}[5m])) by (instance, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(scheduler_volume_scheduling_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}[$__rate_interval])) by (cluster, instance, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{instance}} volume",
+                  "legendFormat": "{{cluster}} {{instance}} volume",
                  "refId": "D"
                }
              ],
@ -3327,18 +3850,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 5,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3362,28 +3888,28 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"2..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"2..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "2xx",
                  "refId": "A"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"3..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"3..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "3xx",
                  "refId": "B"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"4..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"4..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "4xx",
                  "refId": "C"
                },
                {
-                  "expr": "sum(rate(rest_client_requests_total{job=\"kube-scheduler\", instance=~\"$instance\",code=~\"5..\"}[5m]))",
+                  "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\",code=~\"5..\"}[$__rate_interval]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "5xx",
@ -3439,18 +3965,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 6,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3474,7 +4003,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{job=\"kube-scheduler\", instance=~\"$instance\", verb=\"POST\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\", verb=\"POST\"}[$__rate_interval])) by (verb, url, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{verb}} {{url}}",
@ -3543,10 +4072,12 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 7,
+              "interval": "1m",
              "legend": {
                "alignAsTable": true,
                "avg": false,
@ -3555,6 +4086,7 @@ data:
                "min": false,
                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": true
              },
@ -3578,7 +4110,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{job=\"kube-scheduler\", instance=~\"$instance\", verb=\"GET\"}[5m])) by (verb, url, le))",
+                  "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\", verb=\"GET\"}[$__rate_interval])) by (verb, url, le))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{verb}} {{url}}",
@ -3647,18 +4179,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 8,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3682,7 +4217,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "process_resident_memory_bytes{job=\"kube-scheduler\", instance=~\"$instance\"}",
+                  "expr": "process_resident_memory_bytes{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -3738,18 +4273,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 9,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3773,7 +4311,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(process_cpu_seconds_total{job=\"kube-scheduler\", instance=~\"$instance\"}[5m])",
+                  "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", job=\"kube-scheduler\", instance=~\"$instance\"}[$__rate_interval])",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -3829,18 +4367,21 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
              "id": 10,
+              "interval": "1m",
              "legend": {
-                "alignAsTable": false,
+                "alignAsTable": true,
                "avg": false,
                "current": false,
                "max": false,
                "min": false,
-                "rightSide": false,
+                "rightSide": true,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -3864,7 +4405,7 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "go_goroutines{job=\"kube-scheduler\",instance=~\"$instance\"}",
+                  "expr": "go_goroutines{cluster=\"$cluster\", job=\"kube-scheduler\",instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "{{instance}}",
@ -3934,7 +4475,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -3948,6 +4489,32 @@ data:
            "allValue": null,
            "current": {

+            },
+            "datasource": "$datasource",
+            "hide": 2,
+            "includeAll": false,
+            "label": "cluster",
+            "multi": false,
+            "name": "cluster",
+            "options": [
+
+            ],
+            "query": "label_values(up{job=\"kube-scheduler\"}, cluster)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 1,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
+          {
+            "allValue": null,
+            "current": {
+
            },
            "datasource": "$datasource",
            "hide": 0,
@ -3958,7 +4525,7 @@ data:
            "options": [

            ],
-            "query": "label_values(process_cpu_seconds_total{job=\"kube-scheduler\"}, instance)",
+            "query": "label_values(up{job=\"kube-scheduler\", cluster=\"$cluster\"}, instance)",
            "refresh": 2,
            "regex": "",
            "sort": 1,
@ -4001,7 +4568,7 @@ data:
          "30d"
        ]
      },
-      "timezone": "",
+      "timezone": "UTC",
      "title": "Kubernetes / Scheduler",
      "uid": "2e6b6a3b4bddf1427b3a55aa1311c656",
      "version": 0
--- a/addons/grafana/dashboards-nginx-ingress.yaml
+++ b/addons/grafana/dashboards-nginx-ingress.yaml
@ -172,7 +172,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "sum(avg_over_time(nginx_ingress_controller_nginx_process_connections{cluster=~\"$cluster\", controller_pod=~\"$controller\",controller_class=~\"$controller_class\",controller_namespace=~\"$namespace\"}[2m]))",
+                  "expr": "sum(avg_over_time(nginx_ingress_controller_nginx_process_connections{cluster=~\"$cluster\", controller_pod=~\"$controller\",controller_class=~\"$controller_class\",controller_namespace=~\"$namespace\",state=\"active\"}[2m]))",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -296,6 +296,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -308,6 +309,7 @@ data:
                "min": false,
                "rightSide": "true",
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -387,6 +389,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -399,6 +402,7 @@ data:
                "min": false,
                "rightSide": "true",
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -491,6 +495,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -503,6 +508,7 @@ data:
                "min": false,
                "rightSide": "true",
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -609,6 +615,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -621,6 +628,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -707,6 +715,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -719,6 +728,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
@ -798,6 +808,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -810,6 +821,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": "true",
+                "sideWidth": null,
                "total": false,
                "values": "true"
              },
--- a/addons/grafana/dashboards-node-exporter.yaml
+++ b/addons/grafana/dashboards-node-exporter.yaml
@ -15,13 +15,13 @@ data:
      },
      "editable": false,
      "gnetId": null,
-      "graphTooltip": 0,
+      "graphTooltip": 1,
      "hideControls": false,
      "id": null,
      "links": [

      ],
-      "refresh": "",
+      "refresh": "30s",
      "rows": [
        {
          "collapse": false,
@ -36,6 +36,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -48,6 +49,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -71,9 +73,8 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(\n  (1 - rate(node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"}[$__interval]))\n/ ignoring(cpu) group_left\n  count without (cpu)( node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"})\n)\n",
+                  "expr": "(\n  (1 - sum without (mode) (rate(node_cpu_seconds_total{job=\"node-exporter\", mode=~\"idle|iowait|steal\", instance=\"$instance\"}[$__rate_interval])))\n/ ignoring(cpu) group_left\n  count without (cpu, mode) (node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"})\n)\n",
                  "format": "time_series",
-                  "interval": "1m",
                  "intervalFactor": 5,
                  "legendFormat": "{{cpu}}",
                  "refId": "A"
@ -128,6 +129,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 0,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -140,6 +142,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -253,6 +256,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -265,6 +269,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -417,7 +422,7 @@ data:
              "tableColumn": "",
              "targets": [
                {
-                  "expr": "100 -\n(\n  node_memory_MemAvailable_bytes{job=\"node-exporter\", instance=\"$instance\"}\n/\n  node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"}\n* 100\n)\n",
+                  "expr": "100 -\n(\n  avg(node_memory_MemAvailable_bytes{job=\"node-exporter\", instance=\"$instance\"})\n/\n  avg(node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"})\n* 100\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
                  "legendFormat": "",
@ -459,6 +464,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 0,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -471,6 +477,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -501,25 +508,22 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(node_disk_read_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__interval])",
+                  "expr": "rate(node_disk_read_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__rate_interval])",
                  "format": "time_series",
-                  "interval": "1m",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}} read",
                  "refId": "A"
                },
                {
-                  "expr": "rate(node_disk_written_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__interval])",
+                  "expr": "rate(node_disk_written_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__rate_interval])",
                  "format": "time_series",
-                  "interval": "1m",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}} written",
                  "refId": "B"
                },
                {
-                  "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__interval])",
+                  "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", instance=\"$instance\", device!~\"dm.*\"}[$__rate_interval])",
                  "format": "time_series",
-                  "interval": "1m",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}} io time",
                  "refId": "C"
@ -574,6 +578,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -586,6 +591,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -692,6 +698,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 0,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -704,6 +711,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -727,9 +735,8 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(node_network_receive_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__interval])",
+                  "expr": "rate(node_network_receive_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__rate_interval])",
                  "format": "time_series",
-                  "interval": "1m",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}}",
                  "refId": "A"
@ -784,6 +791,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 0,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -796,6 +804,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -819,9 +828,8 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(node_network_transmit_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__interval])",
+                  "expr": "rate(node_network_transmit_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__rate_interval])",
                  "format": "time_series",
-                  "interval": "1m",
                  "intervalFactor": 2,
                  "legendFormat": "{{device}}",
                  "refId": "A"
@ -880,17 +888,17 @@ data:
      "schemaVersion": 14,
      "style": "dark",
      "tags": [
-
+        "node-exporter-mixin"
      ],
      "templating": {
        "list": [
          {
            "current": {
-              "text": "Prometheus",
-              "value": "Prometheus"
+              "text": "default",
+              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -958,7 +966,7 @@ data:
        ]
      },
      "timezone": "",
-      "title": "Nodes",
+      "title": "Node Exporter / Nodes",
      "uid": "fa49a4706d07a042595b664c87fb33ea",
      "version": 0
    }
--- a/addons/grafana/dashboards-prom.yaml
+++ b/addons/grafana/dashboards-prom.yaml
@ -21,7 +21,7 @@ data:
      "links": [

      ],
-      "refresh": "",
+      "refresh": "60s",
      "rows": [
        {
          "collapse": false,
@ -36,6 +36,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -48,6 +49,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -71,10 +73,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(\n  prometheus_remote_storage_highest_timestamp_in_seconds{cluster=~\"$cluster\", instance=~\"$instance\"} \n-  \n  ignoring(queue) group_right(instance) prometheus_remote_storage_queue_highest_sent_timestamp_seconds{cluster=~\"$cluster\", instance=~\"$instance\"}\n)\n",
+                  "expr": "(\n  prometheus_remote_storage_highest_timestamp_in_seconds{cluster=~\"$cluster\", instance=~\"$instance\"} \n-  \n  ignoring(remote_name, url) group_right(instance) (prometheus_remote_storage_queue_highest_sent_timestamp_seconds{cluster=~\"$cluster\", instance=~\"$instance\"} != 0)\n)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -127,6 +129,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -139,6 +142,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -162,10 +166,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "(\n  rate(prometheus_remote_storage_highest_timestamp_in_seconds{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])  \n- \n  ignoring (queue) group_right(instance) rate(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])\n)\n",
+                  "expr": "clamp_min(\n  rate(prometheus_remote_storage_highest_timestamp_in_seconds{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])  \n- \n  ignoring (remote_name, url) group_right(instance) rate(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])\n, 0)\n",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -231,6 +235,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -243,6 +248,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -266,10 +272,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(\n  prometheus_remote_storage_samples_in_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])\n- \n  ignoring(queue) group_right(instance) rate(prometheus_remote_storage_succeeded_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]) \n- \n  rate(prometheus_remote_storage_dropped_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])\n",
+                  "expr": "rate(\n  prometheus_remote_storage_samples_in_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])\n- \n  ignoring(remote_name, url) group_right(instance) (rate(prometheus_remote_storage_succeeded_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]) or rate(prometheus_remote_storage_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]))\n- \n  (rate(prometheus_remote_storage_dropped_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]) or rate(prometheus_remote_storage_samples_dropped_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]))\n",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -335,6 +341,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -347,6 +354,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -374,7 +382,7 @@ data:
                  "expr": "prometheus_remote_storage_shards{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -427,6 +435,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -439,6 +448,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -465,7 +475,7 @@ data:
                  "expr": "prometheus_remote_storage_shards_max{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -518,6 +528,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -530,6 +541,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -556,7 +568,7 @@ data:
                  "expr": "prometheus_remote_storage_shards_min{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -609,6 +621,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -621,6 +634,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -647,7 +661,7 @@ data:
                  "expr": "prometheus_remote_storage_shards_desired{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -713,6 +727,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -725,6 +740,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -751,7 +767,7 @@ data:
                  "expr": "prometheus_remote_storage_shard_capacity{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -804,6 +820,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -816,6 +833,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -839,10 +857,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "prometheus_remote_storage_pending_samples{cluster=~\"$cluster\", instance=~\"$instance\"}",
+                  "expr": "prometheus_remote_storage_pending_samples{cluster=~\"$cluster\", instance=~\"$instance\"} or prometheus_remote_storage_samples_pending{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -908,6 +926,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -920,6 +939,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -999,6 +1019,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -1011,6 +1032,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1037,7 +1059,7 @@ data:
                  "expr": "prometheus_wal_watcher_current_segment{cluster=~\"$cluster\", instance=~\"$instance\"}",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{consumer}}",
                  "refId": "A"
                }
              ],
@ -1103,6 +1125,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -1115,6 +1138,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1138,10 +1162,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(prometheus_remote_storage_dropped_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
+                  "expr": "rate(prometheus_remote_storage_dropped_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]) or rate(prometheus_remote_storage_samples_dropped_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -1194,6 +1218,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -1206,6 +1231,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1229,10 +1255,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(prometheus_remote_storage_failed_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
+                  "expr": "rate(prometheus_remote_storage_failed_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -1285,6 +1311,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -1297,6 +1324,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1320,10 +1348,10 @@ data:
              "steppedLine": false,
              "targets": [
                {
-                  "expr": "rate(prometheus_remote_storage_retried_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
+                  "expr": "rate(prometheus_remote_storage_retried_samples_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m]) or rate(prometheus_remote_storage_samples_retried_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -1376,6 +1404,7 @@ data:
              "dashes": false,
              "datasource": "$datasource",
              "fill": 1,
+              "fillGradient": 0,
              "gridPos": {

              },
@ -1388,6 +1417,7 @@ data:
                "min": false,
                "rightSide": false,
                "show": true,
+                "sideWidth": null,
                "total": false,
                "values": false
              },
@ -1414,7 +1444,7 @@ data:
                  "expr": "rate(prometheus_remote_storage_enqueue_retries_total{cluster=~\"$cluster\", instance=~\"$instance\"}[5m])",
                  "format": "time_series",
                  "intervalFactor": 2,
-                  "legendFormat": "{{cluster}}:{{instance}}-{{queue}}",
+                  "legendFormat": "{{cluster}}:{{instance}} {{remote_name}}:{{url}}",
                  "refId": "A"
                }
              ],
@ -1471,7 +1501,7 @@ data:
      "schemaVersion": 14,
      "style": "dark",
      "tags": [
-
+        "prometheus-mixin"
      ],
      "templating": {
        "list": [
@ -1487,41 +1517,6 @@ data:
            "regex": "",
            "type": "datasource"
          },
-          {
-            "allValue": null,
-            "current": {
-              "text": {
-                "selected": true,
-                "text": "All",
-                "value": "$__all"
-              },
-              "value": {
-                "selected": true,
-                "text": "All",
-                "value": "$__all"
-              }
-            },
-            "datasource": "$datasource",
-            "hide": 0,
-            "includeAll": true,
-            "label": null,
-            "multi": false,
-            "name": "instance",
-            "options": [
-
-            ],
-            "query": "label_values(prometheus_build_info, instance)",
-            "refresh": 2,
-            "regex": "",
-            "sort": 0,
-            "tagValuesQuery": "",
-            "tags": [
-
-            ],
-            "tagsQuery": "",
-            "type": "query",
-            "useTags": false
-          },
          {
            "allValue": null,
            "current": {
@ -1557,6 +1552,41 @@ data:
            "type": "query",
            "useTags": false
          },
+          {
+            "allValue": null,
+            "current": {
+              "text": {
+                "selected": true,
+                "text": "All",
+                "value": "$__all"
+              },
+              "value": {
+                "selected": true,
+                "text": "All",
+                "value": "$__all"
+              }
+            },
+            "datasource": "$datasource",
+            "hide": 0,
+            "includeAll": true,
+            "label": null,
+            "multi": false,
+            "name": "instance",
+            "options": [
+
+            ],
+            "query": "label_values(prometheus_build_info{cluster=~\"$cluster\"}, instance)",
+            "refresh": 2,
+            "regex": "",
+            "sort": 0,
+            "tagValuesQuery": "",
+            "tags": [
+
+            ],
+            "tagsQuery": "",
+            "type": "query",
+            "useTags": false
+          },
          {
            "allValue": null,
            "current": {
@ -1567,11 +1597,11 @@ data:
            "includeAll": true,
            "label": null,
            "multi": false,
-            "name": "queue",
+            "name": "url",
            "options": [

            ],
-            "query": "label_values(prometheus_remote_storage_shards{cluster=~\"$cluster\", instance=~\"$instance\"}, queue)",
+            "query": "label_values(prometheus_remote_storage_shards{cluster=~\"$cluster\", instance=~\"$instance\"}, url)",
            "refresh": 2,
            "regex": "",
            "sort": 0,
@ -1615,7 +1645,7 @@ data:
        ]
      },
      "timezone": "browser",
-      "title": "Prometheus Remote Write",
+      "title": "Prometheus / Remote Write",
      "version": 0
    }
  prometheus.json: |-
@ -1632,7 +1662,7 @@ data:
      "links": [

      ],
-      "refresh": "10s",
+      "refresh": "60s",
      "rows": [
        {
          "collapse": false,
@ -1690,6 +1720,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #A",
@ -1708,6 +1739,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "Value #B",
@ -1726,6 +1758,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "instance",
@ -1744,6 +1777,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "job",
@ -1762,6 +1796,7 @@ data:
                  "dateFormat": "YYYY-MM-DD HH:mm:ss",
                  "decimals": 2,
                  "link": false,
+                  "linkTargetBlank": false,
                  "linkTooltip": "Drill down",
                  "linkUrl": "",
                  "pattern": "version",
@ -1815,7 +1850,7 @@ data:
              "title": "Prometheus Stats",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "transform": "table",
@ -1914,7 +1949,7 @@ data:
              "title": "Target Sync",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2000,7 +2035,7 @@ data:
              "title": "Targets",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2098,7 +2133,7 @@ data:
              "title": "Average Scrape Interval Duration",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2167,6 +2202,14 @@ data:
              "stack": true,
              "steppedLine": false,
              "targets": [
+                {
+                  "expr": "sum by (job) (rate(prometheus_target_scrapes_exceeded_body_size_limit_total[1m]))",
+                  "format": "time_series",
+                  "intervalFactor": 2,
+                  "legendFormat": "exceeded body size limit: {{job}}",
+                  "legendLink": null,
+                  "step": 10
+                },
                {
                  "expr": "sum by (job) (rate(prometheus_target_scrapes_exceeded_sample_limit_total[1m]))",
                  "format": "time_series",
@ -2208,7 +2251,7 @@ data:
              "title": "Scrape failures",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2294,7 +2337,7 @@ data:
              "title": "Appended Samples",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2392,7 +2435,7 @@ data:
              "title": "Head Series",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2478,7 +2521,7 @@ data:
              "title": "Head Chunks",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2576,7 +2619,7 @@ data:
              "title": "Query Rate",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2662,7 +2705,7 @@ data:
              "title": "Stage Duration",
              "tooltip": {
                "shared": true,
-                "sort": 0,
+                "sort": 2,
                "value_type": "individual"
              },
              "type": "graph",
@ -2706,7 +2749,7 @@ data:
      "schemaVersion": 14,
      "style": "dark",
      "tags": [
-
+        "prometheus-mixin"
      ],
      "templating": {
        "list": [
@ -2716,7 +2759,7 @@ data:
              "value": "default"
            },
            "hide": 0,
-            "label": null,
+            "label": "Data Source",
            "name": "datasource",
            "options": [

@ -2727,7 +2770,7 @@ data:
            "type": "datasource"
          },
          {
-            "allValue": null,
+            "allValue": ".+",
            "current": {
              "selected": true,
              "text": "All",
@ -2742,7 +2785,7 @@ data:
            "options": [

            ],
-            "query": "label_values(prometheus_build_info, job)",
+            "query": "label_values(prometheus_build_info{job=\"prometheus\"}, job)",
            "refresh": 1,
            "regex": "",
            "sort": 2,
@ -2755,7 +2798,7 @@ data:
            "useTags": false
          },
          {
-            "allValue": null,
+            "allValue": ".+",
            "current": {
              "selected": true,
              "text": "All",
@ -2770,7 +2813,7 @@ data:
            "options": [

            ],
-            "query": "label_values(prometheus_build_info, instance)",
+            "query": "label_values(prometheus_build_info{job=~\"$job\"}, instance)",
            "refresh": 1,
            "regex": "",
            "sort": 2,
@ -2814,7 +2857,7 @@ data:
        ]
      },
      "timezone": "utc",
-      "title": "Prometheus",
+      "title": "Prometheus / Overview",
      "uid": "",
      "version": 0
    }
--- a/addons/grafana/deployment.yaml
+++ b/addons/grafana/deployment.yaml
@ -18,12 +18,13 @@ spec:
      labels:
        name: grafana
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: grafana
-          image: docker.io/grafana/grafana:7.0.6
+          image: docker.io/grafana/grafana:9.3.1
          env:
            - name: GF_PATHS_CONFIG
              value: "/etc/grafana/custom.ini"
@ -31,15 +32,22 @@ spec:
            - name: http
              containerPort: 8080
          livenessProbe:
-            httpGet:
-              path: /metrics
+            tcpSocket:
              port: 8080
-            initialDelaySeconds: 10
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 1
+            failureThreshold: 5
+            successThreshold: 1
          readinessProbe:
            httpGet:
-              path: /api/health
+              scheme: HTTP
+              path: /robots.txt
              port: 8080
            initialDelaySeconds: 10
+            periodSeconds: 30
+            successThreshold: 1
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 100m
@ -68,6 +76,8 @@ spec:
              mountPath: /etc/grafana/dashboards/k8s-resources-1
            - name: dashboards-k8s-resources-2
              mountPath: /etc/grafana/dashboards/k8s-resources-2
+            - name: dashboards-k8s-network
+              mountPath: /etc/grafana/dashboards/k8s-network
            - name: dashboards-coredns
              mountPath: /etc/grafana/dashboards/coredns
            - name: dashboards-nginx-ingress
@ -100,6 +110,9 @@ spec:
        - name: dashboards-k8s-resources-1
          configMap:
            name: grafana-dashboards-k8s-resources-1
+        - name: dashboards-k8s-network
+          configMap:
+            name: grafana-dashboards-k8s-network
        - name: dashboards-k8s-resources-2
          configMap:
            name: grafana-dashboards-k8s-resources-2
--- a/addons/nginx-ingress/aws/class.yaml
+++ b/addons/nginx-ingress/aws/class.yaml
@ -1,6 +1,6 @@
-apiVersion: networking.k8s.io/v1beta1
+apiVersion: networking.k8s.io/v1
 kind: IngressClass
 metadata:
  name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public
--- a/addons/nginx-ingress/aws/deployment.yaml
+++ b/addons/nginx-ingress/aws/deployment.yaml
@ -17,14 +17,16 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
          args:
            - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
            - --ingress-class=public
          # use downward API
          env:
@ -47,7 +49,6 @@ spec:
              containerPort: 10254
              hostPort: 10254
          livenessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
@ -55,15 +56,16 @@ spec:
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          readinessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          lifecycle:
            preStop:
--- a/addons/nginx-ingress/aws/rbac/role.yaml
+++ b/addons/nginx-ingress/aws/rbac/role.yaml
@ -10,6 +10,7 @@ rules:
      - configmaps
      - pods
      - secrets
+      - endpoints
    verbs:
      - get
  - apiGroups:
@ -37,3 +38,11 @@ rules:
      - endpoints
    verbs:
      - get
+  - apiGroups:
+      - "coordination.k8s.io"
+    resources:
+      - leases
+    verbs:
+      - create
+      - get
+      - update
--- a/addons/nginx-ingress/azure/class.yaml
+++ b/addons/nginx-ingress/azure/class.yaml
@ -1,6 +1,6 @@
-apiVersion: networking.k8s.io/v1beta1
+apiVersion: networking.k8s.io/v1
 kind: IngressClass
 metadata:
  name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public
--- a/addons/nginx-ingress/azure/deployment.yaml
+++ b/addons/nginx-ingress/azure/deployment.yaml
@ -17,14 +17,16 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
          args:
            - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
            - --ingress-class=public
          # use downward API
          env:
@ -47,7 +49,6 @@ spec:
              containerPort: 10254
              hostPort: 10254
          livenessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
@ -55,15 +56,16 @@ spec:
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          readinessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          lifecycle:
            preStop:
--- a/addons/nginx-ingress/azure/rbac/role.yaml
+++ b/addons/nginx-ingress/azure/rbac/role.yaml
@ -10,6 +10,7 @@ rules:
      - configmaps
      - pods
      - secrets
+      - endpoints
    verbs:
      - get
  - apiGroups:
@ -32,8 +33,11 @@ rules:
    verbs:
      - create
  - apiGroups:
-      - ""
+      - "coordination.k8s.io"
    resources:
-      - endpoints
+      - leases
    verbs:
+      - create
      - get
+      - update
+
--- a/addons/nginx-ingress/bare-metal/class.yaml
+++ b/addons/nginx-ingress/bare-metal/class.yaml
@ -1,6 +1,6 @@
-apiVersion: networking.k8s.io/v1beta1
+apiVersion: networking.k8s.io/v1
 kind: IngressClass
 metadata:
  name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public
--- a/addons/nginx-ingress/bare-metal/deployment.yaml
+++ b/addons/nginx-ingress/bare-metal/deployment.yaml
@ -1,7 +1,7 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: ingress-controller-public
+  name: nginx-ingress-controller
  namespace: ingress
 spec:
  replicas: 2
@ -10,21 +10,23 @@ spec:
      maxUnavailable: 1
  selector:
    matchLabels:
-      name: ingress-controller-public
+      name: nginx-ingress-controller
      phase: prod
  template:
    metadata:
      labels:
-        name: ingress-controller-public
+        name: nginx-ingress-controller
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
          args:
            - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
            - --ingress-class=public
          # use downward API
          env:
@ -76,4 +78,3 @@ spec:
            runAsUser: 101 # www-data
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
-
--- a/addons/nginx-ingress/bare-metal/rbac/role.yaml
+++ b/addons/nginx-ingress/bare-metal/rbac/role.yaml
@ -10,6 +10,7 @@ rules:
      - configmaps
      - pods
      - secrets
+      - endpoints
    verbs:
      - get
  - apiGroups:
@ -32,8 +33,10 @@ rules:
    verbs:
      - create
  - apiGroups:
-      - ""
+      - "coordination.k8s.io"
    resources:
-      - endpoints
+      - leases
    verbs:
+      - create
      - get
+      - update
--- a/addons/nginx-ingress/digital-ocean/class.yaml
+++ b/addons/nginx-ingress/digital-ocean/class.yaml
@ -1,6 +1,6 @@
-apiVersion: networking.k8s.io/v1beta1
+apiVersion: networking.k8s.io/v1
 kind: IngressClass
 metadata:
  name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public
--- a/addons/nginx-ingress/digital-ocean/daemonset.yaml
+++ b/addons/nginx-ingress/digital-ocean/daemonset.yaml
@ -17,14 +17,16 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
          args:
            - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
            - --ingress-class=public
          # use downward API
          env:
@ -47,7 +49,6 @@ spec:
              containerPort: 10254
              hostPort: 10254
          livenessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
@ -55,15 +56,16 @@ spec:
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          readinessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          lifecycle:
            preStop:
--- a/addons/nginx-ingress/digital-ocean/rbac/role.yaml
+++ b/addons/nginx-ingress/digital-ocean/rbac/role.yaml
@ -10,6 +10,7 @@ rules:
      - configmaps
      - pods
      - secrets
+      - endpoints
    verbs:
      - get
  - apiGroups:
@ -32,8 +33,10 @@ rules:
    verbs:
      - create
  - apiGroups:
-      - ""
+      - "coordination.k8s.io"
    resources:
-      - endpoints
+      - leases
    verbs:
+      - create
      - get
+      - update
--- a/addons/nginx-ingress/google-cloud/class.yaml
+++ b/addons/nginx-ingress/google-cloud/class.yaml
@ -1,6 +1,6 @@
-apiVersion: networking.k8s.io/v1beta1
+apiVersion: networking.k8s.io/v1
 kind: IngressClass
 metadata:
  name: public
 spec:
-  controller: k8s.io/ingress-nginx
+  controller: k8s.io/public
--- a/addons/nginx-ingress/google-cloud/deployment.yaml
+++ b/addons/nginx-ingress/google-cloud/deployment.yaml
@ -17,14 +17,16 @@ spec:
      labels:
        name: nginx-ingress-controller
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: nginx-ingress-controller
-          image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1
+          image: registry.k8s.io/ingress-nginx/controller:v1.5.1
          args:
            - /nginx-ingress-controller
+            - --controller-class=k8s.io/public
            - --ingress-class=public
          # use downward API
          env:
@ -47,7 +49,6 @@ spec:
              containerPort: 10254
              hostPort: 10254
          livenessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
@ -55,15 +56,16 @@ spec:
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          readinessProbe:
-            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
+            failureThreshold: 3
            timeoutSeconds: 5
          lifecycle:
            preStop:
--- a/addons/nginx-ingress/google-cloud/rbac/role.yaml
+++ b/addons/nginx-ingress/google-cloud/rbac/role.yaml
@ -10,6 +10,7 @@ rules:
      - configmaps
      - pods
      - secrets
+      - endpoints
    verbs:
      - get
  - apiGroups:
@ -32,8 +33,10 @@ rules:
    verbs:
      - create
  - apiGroups:
-      - ""
+      - "coordination.k8s.io"
    resources:
-      - endpoints
+      - leases
    verbs:
+      - create
      - get
+      - update
--- a/addons/prometheus/config.yaml
+++ b/addons/prometheus/config.yaml
@ -34,7 +34,7 @@ data:
    - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
-      
+
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
@ -68,13 +68,58 @@ data:
      - source_labels: [__name__, group]
        regex: apiserver_request_duration_seconds_bucket;.+
        action: drop
+      - source_labels: [__name__, group]
+        regex: apiserver_request_duration_seconds_count;.+
+        action: drop
+
+    # Scrape config for kube-controller-manager endpoints.
+    #
+    # kube-controller-manager service endpoints can be discovered by using the
+    # `endpoints` role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-controller-manager and the `https` port.
+    - job_name: 'kube-controller-manager'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-controller-manager;metrics
+      - replacement: kube-controller-manager
+        action: replace
+        target_label: job
+
+    # Scrape config for kube-scheduler endpoints.
+    #
+    # kube-scheduler service endpoints can be discovered by using the `endpoints`
+    # role and relabelling to only keep only endpoints associated with
+    # kube-system/kube-scheduler and the `https` port.
+    - job_name: 'kube-scheduler'
+      kubernetes_sd_configs:
+      - role: endpoints
+      scheme: https
+      tls_config:
+        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+        insecure_skip_verify: true
+      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
+        action: keep
+        regex: kube-system;kube-scheduler;metrics
+      - replacement: kube-scheduler
+        action: replace
+        target_label: job

    # Scrape config for node (i.e. kubelet) /metrics (e.g. 'kubelet_'). Explore
    # metrics from a node by scraping kubelet (127.0.0.1:10250/metrics).
    - job_name: 'kubelet'
      kubernetes_sd_configs:
      - role: node
-      
+
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
@ -82,10 +127,6 @@ data:
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

-      relabel_configs:
-      - action: labelmap
-        regex: __meta_kubernetes_node_name
-
    # Scrape config for Kubelet cAdvisor. Explore metrics from a node by
    # scraping kubelet (127.0.0.1:10250/metrics/cadvisor).
    - job_name: 'kubernetes-cadvisor'
@ -100,9 +141,6 @@ data:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

-      relabel_configs:
-      - action: labelmap
-        regex: __meta_kubernetes_node_name
      metric_relabel_configs:
      - source_labels: [__name__, image]
        action: drop
@ -121,13 +159,11 @@ data:
      - source_labels: [__meta_kubernetes_node_label_node_kubernetes_io_controller]
        action: keep
        regex: 'true'
-      - action: labelmap
-        regex: __meta_kubernetes_node_name
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        action: replace
        target_label: __address__
        replacement: '${1}:2381'
-    
+
    # Scrape config for service endpoints.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
@ -139,6 +175,7 @@ data:
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
+    # * `prometheus.io/param`: Custom metrics query parameter, like "format=prometheus".
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
@ -161,6 +198,11 @@ data:
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_param]
+        action: replace
+        target_label: __param_$1
+        regex: ([^=]+)=(.*)
+        replacement: $2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
@ -172,44 +214,12 @@ data:
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: job
-      
+
      metric_relabel_configs:
      - source_labels: [__name__]
        action: drop
        regex: etcd_(debugging|disk|request|server).*

-    # Example scrape config for probing services via the Blackbox Exporter.
-    #
-    # The relabeling allows the actual service scrape endpoint to be configured
-    # via the following annotations:
-    #
-    # * `prometheus.io/probe`: Only probe services that have a value of `true`
-    - job_name: 'kubernetes-services'
-
-      metrics_path: /probe
-      params:
-        module: [http_2xx]
-
-      kubernetes_sd_configs:
-      - role: service
-
-      relabel_configs:
-      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
-        action: keep
-        regex: true
-      - source_labels: [__address__]
-        target_label: __param_target
-      - target_label: __address__
-        replacement: blackbox
-      - source_labels: [__param_target]
-        target_label: instance
-      - action: labelmap
-        regex: __meta_kubernetes_service_label_(.+)
-      - source_labels: [__meta_kubernetes_namespace]
-        target_label: namespace
-      - source_labels: [__meta_kubernetes_service_name]
-        target_label: job
-
    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
@ -246,6 +256,67 @@ data:
        action: replace
        target_label: kubernetes_pod_name

+    # Example scrape config for probing Services via the Blackbox Exporter.
+    #
+    # Relabeling allows service scraping to be configured via annotations:
+    # * `prometheus.io/probe`: Only probe services that have a value of `true`
+    - job_name: 'kubernetes-services'
+
+      metrics_path: /probe
+      params:
+        module: [http_2xx]
+
+      kubernetes_sd_configs:
+      - role: service
+
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
+        action: keep
+        regex: true
+      - source_labels: [__address__]
+        target_label: __param_target
+      - target_label: __address__
+        replacement: blackbox-exporter:8080
+      - source_labels: [__param_target]
+        target_label: instance
+      - action: labelmap
+        regex: __meta_kubernetes_service_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_service_name]
+        target_label: job
+
+    # Example scrape config for probing Ingresses via a Blackbox Exporter.
+    #
+    # Relabeling allows service scraping to be configured via annotations:
+    # * `prometheus.io/probe`: Only probe ingresses that have a value of `true`
+    - job_name: 'kubernetes-ingresses'
+      metrics_path: /probe
+      params:
+        module: [http_2xx]
+
+      kubernetes_sd_configs:
+      - role: ingress
+
+      relabel_configs:
+      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
+        regex: (.+);(.+);(.+)
+        replacement: ${1}://${2}${3}
+        target_label: __param_target
+      - target_label: __address__
+        replacement: blackbox-exporter:8080
+      - source_labels: [__param_target]
+        target_label: instance
+      - action: labelmap
+        regex: __meta_kubernetes_ingress_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_service_name]
+        target_label: job
+
    # Rule files
    rule_files:
      - "/etc/prometheus/rules/*.rules"
--- a/addons/prometheus/deployment.yaml
+++ b/addons/prometheus/deployment.yaml
@ -14,13 +14,14 @@ spec:
      labels:
        name: prometheus
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      serviceAccountName: prometheus
      containers:
        - name: prometheus
-          image: quay.io/prometheus/prometheus:v2.19.2
+          image: quay.io/prometheus/prometheus:v2.40.5
          args:
            - --web.listen-address=0.0.0.0:9090
            - --config.file=/etc/prometheus/prometheus.yaml
--- a/addons/prometheus/discovery/kube-controller-manager.yaml
+++ b/addons/prometheus/discovery/kube-controller-manager.yaml
@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
  name: kube-controller-manager
  namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
  clusterIP: None
@ -14,5 +12,5 @@ spec:
  ports:
    - name: metrics
      protocol: TCP
-      port: 10252
-      targetPort: 10252
+      port: 10257
+      targetPort: 10257
--- a/addons/prometheus/discovery/kube-scheduler.yaml
+++ b/addons/prometheus/discovery/kube-scheduler.yaml
@ -1,11 +1,9 @@
-# Allow Prometheus to scrape service endpoints
+# Allow Prometheus to discover service endpoints
 apiVersion: v1
 kind: Service
 metadata:
  name: kube-scheduler
  namespace: kube-system
-  annotations:
-    prometheus.io/scrape: 'true'
 spec:
  type: ClusterIP
  clusterIP: None
@ -14,5 +12,5 @@ spec:
  ports:
    - name: metrics
      protocol: TCP
-      port: 10251
-      targetPort: 10251
+      port: 10259
+      targetPort: 10259
--- a/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/cluster-role.yaml
@ -78,13 +78,6 @@ rules:
  verbs:
  - list
  - watch
- apiGroups:
-  - autoscaling.k8s.io
-  resources:
-  - verticalpodautoscalers
-  verbs:
-  - list
-  - watch
 - apiGroups:
  - admissionregistration.k8s.io
  resources:
@ -97,6 +90,14 @@ rules:
  - networking.k8s.io
  resources:
  - networkpolicies
+  - ingresses
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - coordination.k8s.io
+  resources:
+  - leases
  verbs:
  - list
  - watch
--- a/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
+++ b/addons/prometheus/exporters/kube-state-metrics/deployment.yaml
@ -18,16 +18,19 @@ spec:
      labels:
        name: kube-state-metrics
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
-        image: quay.io/coreos/kube-state-metrics:v1.9.7
+        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0
        ports:
          - name: metrics
            containerPort: 8080
+          - name: telemetry
+            containerPort: 8081
        livenessProbe:
          httpGet:
            path: /healthz
@ -40,3 +43,5 @@ spec:
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
+        securityContext:
+          runAsUser: 65534
--- a/addons/prometheus/exporters/node-exporter/daemonset.yaml
+++ b/addons/prometheus/exporters/node-exporter/daemonset.yaml
@ -17,24 +17,24 @@ spec:
      labels:
        name: node-exporter
        phase: prod
-      annotations:
-        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      serviceAccountName: node-exporter
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
+        runAsGroup: 65534
+        fsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
-        image: quay.io/prometheus/node-exporter:v1.0.1
+        image: quay.io/prometheus/node-exporter:v1.5.0
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
          - --path.rootfs=/host/root
-          - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
-          - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
        ports:
          - name: metrics
            containerPort: 9100
@ -46,6 +46,9 @@ spec:
          limits:
            cpu: 200m
            memory: 100Mi
+        securityContext:
+          seLinuxOptions:
+            type: spc_t
        volumeMounts:
          - name: proc
            mountPath: /host/proc
@ -55,9 +58,12 @@ spec:
            readOnly: true
          - name: root
            mountPath: /host/root
+            mountPropagation: HostToContainer
            readOnly: true
      tolerations:
-        - key: node-role.kubernetes.io/master
+        - key: node-role.kubernetes.io/controller
+          operator: Exists
+        - key: node-role.kubernetes.io/control-plane
          operator: Exists
        - key: node.kubernetes.io/not-ready
          operator: Exists
--- a/addons/prometheus/rbac/cluster-role.yaml
+++ b/addons/prometheus/rbac/cluster-role.yaml
@ -10,6 +10,17 @@ rules:
  - services
  - endpoints
  - pods
-  verbs: ["get", "list", "watch"]
+  verbs:
+  - get
+  - list
+  - watch
 - nonResourceURLs: ["/metrics"]
  verbs: ["get"]
+- apiGroups:
+  - networking.k8s.io
+  resources:
+  - ingresses
+  verbs:
+  - get
+  - list
+  - watch
--- a/addons/prometheus/rules.yaml
+++ b/addons/prometheus/rules.yaml
@ -9,10 +9,11 @@ data:
            {
              "alert": "etcdMembersDown",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }})."
+                "description": "etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).",
+                "summary": "etcd cluster members are down."
              },
-              "expr": "max by (job) (\n  sum by (job) (up{job=~\".*etcd.*\"} == bool 0)\nor\n  count by (job,endpoint) (\n    sum by (job,endpoint,To) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[3m])) > 0.01\n  )\n)\n> 0\n",
-              "for": "3m",
+              "expr": "max without (endpoint) (\n  sum without (instance) (up{job=~\".*etcd.*\"} == bool 0)\nor\n  count without (To) (\n    sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[120s])) > 0.01\n  )\n)\n> 0\n",
+              "for": "10m",
              "labels": {
                "severity": "critical"
              }
@ -20,9 +21,10 @@ data:
            {
              "alert": "etcdInsufficientMembers",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": insufficient members ({{ $value }})."
+                "description": "etcd cluster \"{{ $labels.job }}\": insufficient members ({{ $value }}).",
+                "summary": "etcd cluster has insufficient number of members."
              },
-              "expr": "sum(up{job=~\".*etcd.*\"} == bool 1) by (job) < ((count(up{job=~\".*etcd.*\"}) by (job) + 1) / 2)\n",
+              "expr": "sum(up{job=~\".*etcd.*\"} == bool 1) without (instance) < ((count(up{job=~\".*etcd.*\"}) without (instance) + 1) / 2)\n",
              "for": "3m",
              "labels": {
                "severity": "critical"
@ -31,7 +33,8 @@ data:
            {
              "alert": "etcdNoLeader",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader."
+                "description": "etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.",
+                "summary": "etcd cluster has no leader."
              },
              "expr": "etcd_server_has_leader{job=~\".*etcd.*\"} == 0\n",
              "for": "1m",
@ -42,9 +45,10 @@ data:
            {
              "alert": "etcdHighNumberOfLeaderChanges",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": {{ $value }} leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated."
+                "description": "etcd cluster \"{{ $labels.job }}\": {{ $value }} leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.",
+                "summary": "etcd cluster has high number of leader changes."
              },
-              "expr": "increase((max by (job) (etcd_server_leader_changes_seen_total{job=~\".*etcd.*\"}) or 0*absent(etcd_server_leader_changes_seen_total{job=~\".*etcd.*\"}))[15m:1m]) >= 3\n",
+              "expr": "increase((max without (instance) (etcd_server_leader_changes_seen_total{job=~\".*etcd.*\"}) or 0*absent(etcd_server_leader_changes_seen_total{job=~\".*etcd.*\"}))[15m:1m]) >= 4\n",
              "for": "5m",
              "labels": {
                "severity": "warning"
@ -53,9 +57,10 @@ data:
            {
              "alert": "etcdGRPCRequestsSlow",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": gRPC requests to {{ $labels.grpc_method }} are taking {{ $value }}s on etcd instance {{ $labels.instance }}."
+                "description": "etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.",
+                "summary": "etcd grpc requests are slow"
              },
-              "expr": "histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~\".*etcd.*\", grpc_type=\"unary\"}[5m])) by (job, instance, grpc_service, grpc_method, le))\n> 0.15\n",
+              "expr": "histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~\".*etcd.*\", grpc_method!=\"Defragment\", grpc_type=\"unary\"}[5m])) without(grpc_type))\n> 0.15\n",
              "for": "10m",
              "labels": {
                "severity": "critical"
@ -64,7 +69,8 @@ data:
            {
              "alert": "etcdMemberCommunicationSlow",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}."
+                "description": "etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.",
+                "summary": "etcd cluster member communication is slow."
              },
              "expr": "histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.15\n",
              "for": "10m",
@ -75,7 +81,8 @@ data:
            {
              "alert": "etcdHighNumberOfFailedProposals",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}."
+                "description": "etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.",
+                "summary": "etcd cluster has high number of proposal failures."
              },
              "expr": "rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) > 5\n",
              "for": "15m",
@ -86,7 +93,8 @@ data:
            {
              "alert": "etcdHighFsyncDurations",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile fync durations are {{ $value }}s on etcd instance {{ $labels.instance }}."
+                "description": "etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.",
+                "summary": "etcd cluster 99th percentile fsync durations are too high."
              },
              "expr": "histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.5\n",
              "for": "10m",
@ -94,10 +102,23 @@ data:
                "severity": "warning"
              }
            },
+            {
+              "alert": "etcdHighFsyncDurations",
+              "annotations": {
+                "description": "etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.",
+                "summary": "etcd cluster 99th percentile fsync durations are too high."
+              },
+              "expr": "histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 1\n",
+              "for": "10m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
            {
              "alert": "etcdHighCommitDurations",
              "annotations": {
-                "message": "etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}."
+                "description": "etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.",
+                "summary": "etcd cluster 99th percentile commit durations are too high."
              },
              "expr": "histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m]))\n> 0.25\n",
              "for": "10m",
@ -106,33 +127,24 @@ data:
              }
            },
            {
-              "alert": "etcdHighNumberOfFailedHTTPRequests",
+              "alert": "etcdBackendQuotaLowSpace",
              "annotations": {
-                "message": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}"
+                "description": "etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.",
+                "summary": "etcd cluster database is running full."
              },
-              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nBY (method) > 0.01\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
-            {
-              "alert": "etcdHighNumberOfFailedHTTPRequests",
-              "annotations": {
-                "message": "{{ $value }}% of requests for {{ $labels.method }} failed on etcd instance {{ $labels.instance }}."
-              },
-              "expr": "sum(rate(etcd_http_failed_total{job=~\".*etcd.*\", code!=\"404\"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~\".*etcd.*\"}[5m]))\nBY (method) > 0.05\n",
+              "expr": "(etcd_mvcc_db_total_size_in_bytes/etcd_server_quota_backend_bytes)*100 > 95\n",
              "for": "10m",
              "labels": {
                "severity": "critical"
              }
            },
            {
-              "alert": "etcdHTTPRequestsSlow",
+              "alert": "etcdExcessiveDatabaseGrowth",
              "annotations": {
-                "message": "etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method }} are slow."
+                "description": "etcd cluster \"{{ $labels.job }}\": Observed surge in etcd writes leading to 50% increase in database size over the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.",
+                "summary": "etcd cluster database growing very fast."
              },
-              "expr": "histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))\n> 0.15\n",
+              "expr": "increase(((etcd_mvcc_db_total_size_in_bytes/etcd_server_quota_backend_bytes)*100)[240m:1m]) > 50\n",
              "for": "10m",
              "labels": {
                "severity": "warning"
@ -146,135 +158,217 @@ data:
    {
      "groups": [
        {
-          "name": "kube-apiserver-error",
+          "name": "kube-apiserver-burnrate.rules",
          "rules": [
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[5m]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[1d]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[1d]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[1d]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[1d]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[1d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1d]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate5m"
+              "record": "apiserver_request:burnrate1d"
            },
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[30m]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[1h]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[1h]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[1h]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[1h]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[1h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate30m"
+              "record": "apiserver_request:burnrate1h"
            },
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[1h]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[2h]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[2h]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[2h]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[2h]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[2h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[2h]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate1h"
+              "record": "apiserver_request:burnrate2h"
            },
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[2h]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[30m]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[30m]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[30m]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[30m]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[30m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate2h"
+              "record": "apiserver_request:burnrate30m"
            },
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[6h]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[3d]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[3d]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[3d]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[3d]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[3d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[3d]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate6h"
+              "record": "apiserver_request:burnrate3d"
            },
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[1d]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[5m]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[5m]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[5m]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[5m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate1d"
+              "record": "apiserver_request:burnrate5m"
            },
            {
-              "expr": "sum by (status_class) (\n  label_replace(\n    rate(apiserver_request_total{job=\"apiserver\"}[3d]\n  ), \"status_class\", \"${1}xx\", \"code\", \"([0-9])..\")\n)\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[6h]))\n    -\n    (\n      (\n        sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=~\"resource|\",le=\"1\"}[6h]))\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"namespace\",le=\"5\"}[6h]))\n      +\n      sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\",scope=\"cluster\",le=\"30\"}[6h]))\n    )\n  )\n  +\n  # errors\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\",code=~\"5..\"}[6h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "read"
              },
-              "record": "status_class:apiserver_request_total:rate3d"
+              "record": "apiserver_request:burnrate6h"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate5m{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate5m{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[1d]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[1d]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1d]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate5m"
+              "record": "apiserver_request:burnrate1d"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate30m{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate30m{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[1h]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[1h]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate30m"
+              "record": "apiserver_request:burnrate1h"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate1h{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate1h{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[2h]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[2h]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[2h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[2h]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate1h"
+              "record": "apiserver_request:burnrate2h"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate2h{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate2h{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[30m]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[30m]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[30m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate2h"
+              "record": "apiserver_request:burnrate30m"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate6h{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate6h{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[3d]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[3d]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[3d]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[3d]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate6h"
+              "record": "apiserver_request:burnrate3d"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate1d{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate1d{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[5m]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[5m]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate1d"
+              "record": "apiserver_request:burnrate5m"
            },
            {
-              "expr": "sum(status_class:apiserver_request_total:rate3d{job=\"apiserver\",status_class=\"5xx\"})\n/\nsum(status_class:apiserver_request_total:rate3d{job=\"apiserver\"})\n",
+              "expr": "(\n  (\n    # too slow\n    sum by (cluster) (rate(apiserver_request_duration_seconds_count{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[6h]))\n    -\n    sum by (cluster) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\",le=\"1\"}[6h]))\n  )\n  +\n  sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",code=~\"5..\"}[6h]))\n)\n/\nsum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))\n",
              "labels": {
-                "job": "apiserver"
+                "verb": "write"
              },
-              "record": "status_class_5xx:apiserver_request_total:ratio_rate3d"
+              "record": "apiserver_request:burnrate6h"
            }
          ]
        },
        {
-          "name": "kube-apiserver.rules",
+          "name": "kube-apiserver-histogram.rules",
          "rules": [
            {
-              "expr": "sum(rate(apiserver_request_duration_seconds_sum{subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod)\n/\nsum(rate(apiserver_request_duration_seconds_count{subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod)\n",
-              "record": "cluster:apiserver_request_duration_seconds:mean5m"
-            },
-            {
-              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
+              "expr": "histogram_quantile(0.99, sum by (cluster, le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))) > 0\n",
              "labels": {
-                "quantile": "0.99"
+                "quantile": "0.99",
+                "verb": "read"
              },
              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
            },
            {
-              "expr": "histogram_quantile(0.9, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
+              "expr": "histogram_quantile(0.99, sum by (cluster, le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\",subresource!~\"proxy|attach|log|exec|portforward\"}[5m]))) > 0\n",
              "labels": {
-                "quantile": "0.9"
+                "quantile": "0.99",
+                "verb": "write"
              },
              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
+            }
+          ]
+        },
+        {
+          "interval": "3m",
+          "name": "kube-apiserver-availability.rules",
+          "rules": [
+            {
+              "expr": "avg_over_time(code_verb:apiserver_request_total:increase1h[30d]) * 24 * 30\n",
+              "record": "code_verb:apiserver_request_total:increase30d"
            },
            {
-              "expr": "histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])) without(instance, pod))\n",
+              "expr": "sum by (cluster, code) (code_verb:apiserver_request_total:increase30d{verb=~\"LIST|GET\"})\n",
              "labels": {
-                "quantile": "0.5"
+                "verb": "read"
              },
-              "record": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile"
+              "record": "code:apiserver_request_total:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, code) (code_verb:apiserver_request_total:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n",
+              "labels": {
+                "verb": "write"
+              },
+              "record": "code:apiserver_request_total:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope) (increase(apiserver_request_duration_seconds_count[1h]))\n",
+              "record": "cluster_verb_scope:apiserver_request_duration_seconds_count:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope) (avg_over_time(cluster_verb_scope:apiserver_request_duration_seconds_count:increase1h[30d]) * 24 * 30)\n",
+              "record": "cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope, le) (increase(apiserver_request_duration_seconds_bucket[1h]))\n",
+              "record": "cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, verb, scope, le) (avg_over_time(cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase1h[30d]) * 24 * 30)\n",
+              "record": "cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d"
+            },
+            {
+              "expr": "1 - (\n  (\n    # write too slow\n    sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n    -\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"})\n  ) +\n  (\n    # read too slow\n    sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"LIST|GET\"})\n    -\n    (\n      (\n        sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=~\"resource|\",le=\"1\"})\n        or\n        vector(0)\n      )\n      +\n      sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"namespace\",le=\"5\"})\n      +\n      sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"cluster\",le=\"30\"})\n    )\n  ) +\n  # errors\n  sum by (cluster) (code:apiserver_request_total:increase30d{code=~\"5..\"} or vector(0))\n)\n/\nsum by (cluster) (code:apiserver_request_total:increase30d)\n",
+              "labels": {
+                "verb": "all"
+              },
+              "record": "apiserver_request:availability30d"
+            },
+            {
+              "expr": "1 - (\n  sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"LIST|GET\"})\n  -\n  (\n    # too slow\n    (\n      sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=~\"resource|\",le=\"1\"})\n      or\n      vector(0)\n    )\n    +\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"namespace\",le=\"5\"})\n    +\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"LIST|GET\",scope=\"cluster\",le=\"30\"})\n  )\n  +\n  # errors\n  sum by (cluster) (code:apiserver_request_total:increase30d{verb=\"read\",code=~\"5..\"} or vector(0))\n)\n/\nsum by (cluster) (code:apiserver_request_total:increase30d{verb=\"read\"})\n",
+              "labels": {
+                "verb": "read"
+              },
+              "record": "apiserver_request:availability30d"
+            },
+            {
+              "expr": "1 - (\n  (\n    # too slow\n    sum by (cluster) (cluster_verb_scope:apiserver_request_duration_seconds_count:increase30d{verb=~\"POST|PUT|PATCH|DELETE\"})\n    -\n    sum by (cluster) (cluster_verb_scope_le:apiserver_request_duration_seconds_bucket:increase30d{verb=~\"POST|PUT|PATCH|DELETE\",le=\"1\"})\n  )\n  +\n  # errors\n  sum by (cluster) (code:apiserver_request_total:increase30d{verb=\"write\",code=~\"5..\"} or vector(0))\n)\n/\nsum by (cluster) (code:apiserver_request_total:increase30d{verb=\"write\"})\n",
+              "labels": {
+                "verb": "write"
+              },
+              "record": "apiserver_request:availability30d"
+            },
+            {
+              "expr": "sum by (cluster,code,resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))\n",
+              "labels": {
+                "verb": "read"
+              },
+              "record": "code_resource:apiserver_request_total:rate5m"
+            },
+            {
+              "expr": "sum by (cluster,code,resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))\n",
+              "labels": {
+                "verb": "write"
+              },
+              "record": "code_resource:apiserver_request_total:rate5m"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"2..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"3..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"4..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
+            },
+            {
+              "expr": "sum by (cluster, code, verb) (increase(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET|POST|PUT|PATCH|DELETE\",code=~\"5..\"}[1h]))\n",
+              "record": "code_verb:apiserver_request_total:increase1h"
            }
          ]
        },
@ -282,61 +376,84 @@ data:
          "name": "k8s.rules",
          "rules": [
            {
-              "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container!=\"POD\"}[5m])) by (namespace)\n",
-              "record": "namespace:container_cpu_usage_seconds_total:sum_rate"
+              "expr": "sum by (cluster, namespace, pod, container) (\n  irate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\"}[5m])\n) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (\n  1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
+              "record": "node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate"
            },
            {
-              "expr": "sum by (cluster, namespace, pod, container) (\n  rate(container_cpu_usage_seconds_total{job=\"kubernetes-cadvisor\", image!=\"\", container!=\"POD\"}[5m])\n) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (\n  1, max by(cluster, namespace, pod, node) (kube_pod_info)\n)\n",
-              "record": "node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate"
-            },
-            {
-              "expr": "container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info)\n)\n",
+              "expr": "container_memory_working_set_bytes{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
              "record": "node_namespace_pod_container:container_memory_working_set_bytes"
            },
            {
-              "expr": "container_memory_rss{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info)\n)\n",
+              "expr": "container_memory_rss{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
              "record": "node_namespace_pod_container:container_memory_rss"
            },
            {
-              "expr": "container_memory_cache{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info)\n)\n",
+              "expr": "container_memory_cache{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
              "record": "node_namespace_pod_container:container_memory_cache"
            },
            {
-              "expr": "container_memory_swap{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info)\n)\n",
+              "expr": "container_memory_swap{job=\"kubernetes-cadvisor\", image!=\"\"}\n* on (namespace, pod) group_left(node) topk by(namespace, pod) (1,\n  max by(namespace, pod, node) (kube_pod_info{node!=\"\"})\n)\n",
              "record": "node_namespace_pod_container:container_memory_swap"
            },
            {
-              "expr": "sum(container_memory_usage_bytes{job=\"kubernetes-cadvisor\", image!=\"\", container!=\"POD\"}) by (namespace)\n",
-              "record": "namespace:container_memory_usage_bytes:sum"
+              "expr": "kube_pod_container_resource_requests{resource=\"memory\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n  (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n)\n",
+              "record": "cluster:namespace:pod_memory:active:kube_pod_container_resource_requests"
            },
            {
-              "expr": "sum by (namespace) (\n    sum by (namespace, pod) (\n        max by (namespace, pod, container) (\n            kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"}\n        ) * on(namespace, pod) group_left() max by (namespace, pod) (\n            kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
-              "record": "namespace:kube_pod_container_resource_requests_memory_bytes:sum"
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_requests{resource=\"memory\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_memory:kube_pod_container_resource_requests:sum"
            },
            {
-              "expr": "sum by (namespace) (\n    sum by (namespace, pod) (\n        max by (namespace, pod, container) (\n            kube_pod_container_resource_requests_cpu_cores{job=\"kube-state-metrics\"}\n        ) * on(namespace, pod) group_left() max by (namespace, pod) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
-              "record": "namespace:kube_pod_container_resource_requests_cpu_cores:sum"
+              "expr": "kube_pod_container_resource_requests{resource=\"cpu\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n  (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n)\n",
+              "record": "cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests"
+            },
+            {
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_requests{resource=\"cpu\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_cpu:kube_pod_container_resource_requests:sum"
+            },
+            {
+              "expr": "kube_pod_container_resource_limits{resource=\"memory\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n  (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n)\n",
+              "record": "cluster:namespace:pod_memory:active:kube_pod_container_resource_limits"
+            },
+            {
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_limits{resource=\"memory\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_memory:kube_pod_container_resource_limits:sum"
+            },
+            {
+              "expr": "kube_pod_container_resource_limits{resource=\"cpu\",job=\"kube-state-metrics\"}  * on (namespace, pod, cluster)\ngroup_left() max by (namespace, pod, cluster) (\n (kube_pod_status_phase{phase=~\"Pending|Running\"} == 1)\n )\n",
+              "record": "cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits"
+            },
+            {
+              "expr": "sum by (namespace, cluster) (\n    sum by (namespace, pod, cluster) (\n        max by (namespace, pod, container, cluster) (\n          kube_pod_container_resource_limits{resource=\"cpu\",job=\"kube-state-metrics\"}\n        ) * on(namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (\n          kube_pod_status_phase{phase=~\"Pending|Running\"} == 1\n        )\n    )\n)\n",
+              "record": "namespace_cpu:kube_pod_container_resource_limits:sum"
            },
            {
              "expr": "max by (cluster, namespace, workload, pod) (\n  label_replace(\n    label_replace(\n      kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"ReplicaSet\"},\n      \"replicaset\", \"$1\", \"owner_name\", \"(.*)\"\n    ) * on(replicaset, namespace) group_left(owner_name) topk by(replicaset, namespace) (\n      1, max by (replicaset, namespace, owner_name) (\n        kube_replicaset_owner{job=\"kube-state-metrics\"}\n      )\n    ),\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n)\n",
              "labels": {
                "workload_type": "deployment"
              },
-              "record": "mixin_pod_workload"
+              "record": "namespace_workload_pod:kube_pod_owner:relabel"
            },
            {
              "expr": "max by (cluster, namespace, workload, pod) (\n  label_replace(\n    kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"DaemonSet\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n)\n",
              "labels": {
                "workload_type": "daemonset"
              },
-              "record": "mixin_pod_workload"
+              "record": "namespace_workload_pod:kube_pod_owner:relabel"
            },
            {
              "expr": "max by (cluster, namespace, workload, pod) (\n  label_replace(\n    kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"StatefulSet\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n)\n",
              "labels": {
                "workload_type": "statefulset"
              },
-              "record": "mixin_pod_workload"
+              "record": "namespace_workload_pod:kube_pod_owner:relabel"
+            },
+            {
+              "expr": "max by (cluster, namespace, workload, pod) (\n  label_replace(\n    kube_pod_owner{job=\"kube-state-metrics\", owner_kind=\"Job\"},\n    \"workload\", \"$1\", \"owner_name\", \"(.*)\"\n  )\n)\n",
+              "labels": {
+                "workload_type": "job"
+              },
+              "record": "namespace_workload_pod:kube_pod_owner:relabel"
            }
          ]
        },
@ -412,20 +529,20 @@ data:
          "name": "node.rules",
          "rules": [
            {
-              "expr": "sum(min(kube_pod_info) by (cluster, node))\n",
-              "record": ":kube_pod_info_node_count:"
-            },
-            {
-              "expr": "topk by(namespace, pod) (1,\n  max by (node, namespace, pod) (\n    label_replace(kube_pod_info{job=\"kube-state-metrics\"}, \"pod\", \"$1\", \"pod\", \"(.*)\")\n))\n",
+              "expr": "topk by(namespace, pod) (1,\n  max by (node, namespace, pod) (\n    label_replace(kube_pod_info{job=\"kube-state-metrics\",node!=\"\"}, \"pod\", \"$1\", \"pod\", \"(.*)\")\n))\n",
              "record": "node_namespace_pod:kube_pod_info:"
            },
            {
-              "expr": "count by (cluster, node) (sum by (node, cpu) (\n  node_cpu_seconds_total{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  node_namespace_pod:kube_pod_info:\n))\n",
+              "expr": "count by (cluster, node) (sum by (node, cpu) (\n  node_cpu_seconds_total{job=\"node-exporter\"}\n* on (namespace, pod) group_left(node)\n  topk by(namespace, pod) (1, node_namespace_pod:kube_pod_info:)\n))\n",
              "record": "node:node_num_cpu:sum"
            },
            {
              "expr": "sum(\n  node_memory_MemAvailable_bytes{job=\"node-exporter\"} or\n  (\n    node_memory_Buffers_bytes{job=\"node-exporter\"} +\n    node_memory_Cached_bytes{job=\"node-exporter\"} +\n    node_memory_MemFree_bytes{job=\"node-exporter\"} +\n    node_memory_Slab_bytes{job=\"node-exporter\"}\n  )\n) by (cluster)\n",
              "record": ":node_memory_MemAvailable_bytes:sum"
+            },
+            {
+              "expr": "sum(rate(node_cpu_seconds_total{job=\"node-exporter\",mode!=\"idle\",mode!=\"iowait\",mode!=\"steal\"}[5m])) /\ncount(sum(node_cpu_seconds_total{job=\"node-exporter\"}) by (cluster, instance, cpu))\n",
+              "record": "cluster:node_cpu:ratio_rate5m"
            }
          ]
        },
@ -433,21 +550,21 @@ data:
          "name": "kubelet.rules",
          "rules": [
            {
-              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (cluster, instance, le) * on(cluster, instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
              "labels": {
                "quantile": "0.99"
              },
              "record": "node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile"
            },
            {
-              "expr": "histogram_quantile(0.9, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
+              "expr": "histogram_quantile(0.9, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (cluster, instance, le) * on(cluster, instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
              "labels": {
                "quantile": "0.9"
              },
              "record": "node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile"
            },
            {
-              "expr": "histogram_quantile(0.5, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
+              "expr": "histogram_quantile(0.5, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (cluster, instance, le) * on(cluster, instance) group_left(node) kubelet_node_name{job=\"kubelet\"})\n",
              "labels": {
                "quantile": "0.5"
              },
@ -461,104 +578,113 @@ data:
            {
              "alert": "KubePodCrashLooping",
              "annotations": {
-                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping"
+                "description": "Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \"CrashLoopBackOff\").",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping",
+                "summary": "Pod is crash looping."
              },
-              "expr": "rate(kube_pod_container_status_restarts_total{job=\"kube-state-metrics\"}[15m]) * 60 * 5 > 0\n",
+              "expr": "max_over_time(kube_pod_container_status_waiting_reason{reason=\"CrashLoopBackOff\", job=\"kube-state-metrics\"}[5m]) >= 1\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubePodNotReady",
              "annotations": {
-                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready"
+                "description": "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready",
+                "summary": "Pod has been in a non-ready state for more than 15 minutes."
              },
-              "expr": "sum by (namespace, pod) (max by(namespace, pod) (kube_pod_status_phase{job=\"kube-state-metrics\", phase=~\"Pending|Unknown\"}) * on(namespace, pod) group_left(owner_kind) max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=\"Job\"})) > 0\n",
+              "expr": "sum by (namespace, pod) (\n  max by(namespace, pod) (\n    kube_pod_status_phase{job=\"kube-state-metrics\", phase=~\"Pending|Unknown\"}\n  ) * on(namespace, pod) group_left(owner_kind) topk by(namespace, pod) (\n    1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=\"Job\"})\n  )\n) > 0\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeDeploymentGenerationMismatch",
              "annotations": {
-                "message": "Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentgenerationmismatch"
+                "description": "Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentgenerationmismatch",
+                "summary": "Deployment generation mismatch due to possible roll-back"
              },
              "expr": "kube_deployment_status_observed_generation{job=\"kube-state-metrics\"}\n  !=\nkube_deployment_metadata_generation{job=\"kube-state-metrics\"}\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeDeploymentReplicasMismatch",
              "annotations": {
-                "message": "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch"
+                "description": "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch",
+                "summary": "Deployment has not matched the expected number of replicas."
              },
-              "expr": "(\n  kube_deployment_spec_replicas{job=\"kube-state-metrics\"}\n    !=\n  kube_deployment_status_replicas_available{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_deployment_status_replicas_updated{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
+              "expr": "(\n  kube_deployment_spec_replicas{job=\"kube-state-metrics\"}\n    >\n  kube_deployment_status_replicas_available{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_deployment_status_replicas_updated{job=\"kube-state-metrics\"}[10m])\n    ==\n  0\n)\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeStatefulSetReplicasMismatch",
              "annotations": {
-                "message": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch"
+                "description": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch",
+                "summary": "Deployment has not matched the expected number of replicas."
              },
-              "expr": "(\n  kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
+              "expr": "(\n  kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas{job=\"kube-state-metrics\"}\n) and (\n  changes(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}[10m])\n    ==\n  0\n)\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeStatefulSetGenerationMismatch",
              "annotations": {
-                "message": "StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetgenerationmismatch"
+                "description": "StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetgenerationmismatch",
+                "summary": "StatefulSet generation mismatch due to possible roll-back"
              },
              "expr": "kube_statefulset_status_observed_generation{job=\"kube-state-metrics\"}\n  !=\nkube_statefulset_metadata_generation{job=\"kube-state-metrics\"}\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeStatefulSetUpdateNotRolledOut",
              "annotations": {
-                "message": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetupdatenotrolledout"
+                "description": "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetupdatenotrolledout",
+                "summary": "StatefulSet update has not been rolled out."
              },
-              "expr": "max without (revision) (\n  kube_statefulset_status_current_revision{job=\"kube-state-metrics\"}\n    unless\n  kube_statefulset_status_update_revision{job=\"kube-state-metrics\"}\n)\n  *\n(\n  kube_statefulset_replicas{job=\"kube-state-metrics\"}\n    !=\n  kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}\n)\n",
+              "expr": "(\n  max without (revision) (\n    kube_statefulset_status_current_revision{job=\"kube-state-metrics\"}\n      unless\n    kube_statefulset_status_update_revision{job=\"kube-state-metrics\"}\n  )\n    *\n  (\n    kube_statefulset_replicas{job=\"kube-state-metrics\"}\n      !=\n    kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}\n  )\n)  and (\n  changes(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeDaemonSetRolloutStuck",
              "annotations": {
-                "message": "Only {{ $value | humanizePercentage }} of the desired Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are scheduled and ready.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetrolloutstuck"
+                "description": "DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetrolloutstuck",
+                "summary": "DaemonSet rollout is stuck."
              },
-              "expr": "kube_daemonset_status_number_ready{job=\"kube-state-metrics\"}\n  /\nkube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"} < 1.00\n",
+              "expr": "(\n  (\n    kube_daemonset_status_current_number_scheduled{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  ) or (\n    kube_daemonset_status_number_misscheduled{job=\"kube-state-metrics\"}\n     !=\n    0\n  ) or (\n    kube_daemonset_status_updated_number_scheduled{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  ) or (\n    kube_daemonset_status_number_available{job=\"kube-state-metrics\"}\n     !=\n    kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  )\n) and (\n  changes(kube_daemonset_status_updated_number_scheduled{job=\"kube-state-metrics\"}[5m])\n    ==\n  0\n)\n",
              "for": "15m",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubeContainerWaiting",
              "annotations": {
-                "message": "Pod {{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container}} has been in waiting state for longer than 1 hour.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontainerwaiting"
+                "description": "pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontainerwaiting",
+                "summary": "Pod container waiting longer than 1 hour"
              },
              "expr": "sum by (namespace, pod, container) (kube_pod_container_status_waiting_reason{job=\"kube-state-metrics\"}) > 0\n",
              "for": "1h",
@ -569,8 +695,9 @@ data:
            {
              "alert": "KubeDaemonSetNotScheduled",
              "annotations": {
-                "message": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetnotscheduled"
+                "description": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetnotscheduled",
+                "summary": "DaemonSet pods are not scheduled."
              },
              "expr": "kube_daemonset_status_desired_number_scheduled{job=\"kube-state-metrics\"}\n  -\nkube_daemonset_status_current_number_scheduled{job=\"kube-state-metrics\"} > 0\n",
              "for": "10m",
@ -581,23 +708,12 @@ data:
            {
              "alert": "KubeDaemonSetMisScheduled",
              "annotations": {
-                "message": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetmisscheduled"
+                "description": "{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetmisscheduled",
+                "summary": "DaemonSet pods are misscheduled."
              },
              "expr": "kube_daemonset_status_number_misscheduled{job=\"kube-state-metrics\"} > 0\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
-            {
-              "alert": "KubeCronJobRunning",
-              "annotations": {
-                "message": "CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} is taking more than 1h to complete.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecronjobrunning"
-              },
-              "expr": "time() - kube_cronjob_next_schedule_time{job=\"kube-state-metrics\"} > 3600\n",
-              "for": "1h",
+              "for": "15m",
              "labels": {
                "severity": "warning"
              }
@ -605,11 +721,12 @@ data:
            {
              "alert": "KubeJobCompletion",
              "annotations": {
-                "message": "Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than one hour to complete.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion"
+                "description": "Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than 12 hours to complete.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion",
+                "summary": "Job did not complete in time"
              },
              "expr": "kube_job_spec_completions{job=\"kube-state-metrics\"} - kube_job_status_succeeded{job=\"kube-state-metrics\"}  > 0\n",
-              "for": "1h",
+              "for": "12h",
              "labels": {
                "severity": "warning"
              }
@ -617,8 +734,9 @@ data:
            {
              "alert": "KubeJobFailed",
              "annotations": {
-                "message": "Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobfailed"
+                "description": "Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobfailed",
+                "summary": "Job failed to complete."
              },
              "expr": "kube_job_failed{job=\"kube-state-metrics\"}  > 0\n",
              "for": "15m",
@ -629,10 +747,11 @@ data:
            {
              "alert": "KubeHpaReplicasMismatch",
              "annotations": {
-                "message": "HPA {{ $labels.namespace }}/{{ $labels.hpa }} has not matched the desired number of replicas for longer than 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubehpareplicasmismatch"
+                "description": "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubehpareplicasmismatch",
+                "summary": "HPA has not matched descired number of replicas."
              },
-              "expr": "(kube_hpa_status_desired_replicas{job=\"kube-state-metrics\"}\n  !=\nkube_hpa_status_current_replicas{job=\"kube-state-metrics\"})\n  and\nchanges(kube_hpa_status_current_replicas[15m]) == 0\n",
+              "expr": "(kube_horizontalpodautoscaler_status_desired_replicas{job=\"kube-state-metrics\"}\n  !=\nkube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"})\n  and\n(kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}\n  >\nkube_horizontalpodautoscaler_spec_min_replicas{job=\"kube-state-metrics\"})\n  and\n(kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}\n  <\nkube_horizontalpodautoscaler_spec_max_replicas{job=\"kube-state-metrics\"})\n  and\nchanges(kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}[15m]) == 0\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -641,10 +760,11 @@ data:
            {
              "alert": "KubeHpaMaxedOut",
              "annotations": {
-                "message": "HPA {{ $labels.namespace }}/{{ $labels.hpa }} has been running at max replicas for longer than 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubehpamaxedout"
+                "description": "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubehpamaxedout",
+                "summary": "HPA is running at max replicas"
              },
-              "expr": "kube_hpa_status_current_replicas{job=\"kube-state-metrics\"}\n  ==\nkube_hpa_spec_max_replicas{job=\"kube-state-metrics\"}\n",
+              "expr": "kube_horizontalpodautoscaler_status_current_replicas{job=\"kube-state-metrics\"}\n  ==\nkube_horizontalpodautoscaler_spec_max_replicas{job=\"kube-state-metrics\"}\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -658,58 +778,89 @@ data:
            {
              "alert": "KubeCPUOvercommit",
              "annotations": {
-                "message": "Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
+                "description": "Cluster has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit",
+                "summary": "Cluster has overcommitted CPU resource requests."
              },
-              "expr": "sum(namespace:kube_pod_container_resource_requests_cpu_cores:sum{})\n  /\nsum(kube_node_status_allocatable_cpu_cores)\n  >\n(count(kube_node_status_allocatable_cpu_cores)-1) / count(kube_node_status_allocatable_cpu_cores)\n",
+              "expr": "sum(namespace_cpu:kube_pod_container_resource_requests:sum{}) - (sum(kube_node_status_allocatable{resource=\"cpu\"}) - max(kube_node_status_allocatable{resource=\"cpu\"})) > 0\nand\n(sum(kube_node_status_allocatable{resource=\"cpu\"}) - max(kube_node_status_allocatable{resource=\"cpu\"})) > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeMemoryOvercommit",
+              "annotations": {
+                "description": "Cluster has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememoryovercommit",
+                "summary": "Cluster has overcommitted memory resource requests."
+              },
+              "expr": "sum(namespace_memory:kube_pod_container_resource_requests:sum{}) - (sum(kube_node_status_allocatable{resource=\"memory\"}) - max(kube_node_status_allocatable{resource=\"memory\"})) > 0\nand\n(sum(kube_node_status_allocatable{resource=\"memory\"}) - max(kube_node_status_allocatable{resource=\"memory\"})) > 0\n",
+              "for": "10m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeCPUQuotaOvercommit",
+              "annotations": {
+                "description": "Cluster has overcommitted CPU resource requests for Namespaces.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuquotaovercommit",
+                "summary": "Cluster has overcommitted CPU resource requests."
+              },
+              "expr": "sum(min without(resource) (kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=~\"(cpu|requests.cpu)\"}))\n  /\nsum(kube_node_status_allocatable{resource=\"cpu\", job=\"kube-state-metrics\"})\n  > 1.5\n",
              "for": "5m",
              "labels": {
                "severity": "warning"
              }
            },
            {
-              "alert": "KubeMemOvercommit",
+              "alert": "KubeMemoryQuotaOvercommit",
              "annotations": {
-                "message": "Cluster has overcommitted memory resource requests for Pods and cannot tolerate node failure.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
+                "description": "Cluster has overcommitted memory resource requests for Namespaces.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememoryquotaovercommit",
+                "summary": "Cluster has overcommitted memory resource requests."
              },
-              "expr": "sum(namespace:kube_pod_container_resource_requests_memory_bytes:sum{})\n  /\nsum(kube_node_status_allocatable_memory_bytes)\n  >\n(count(kube_node_status_allocatable_memory_bytes)-1)\n  /\ncount(kube_node_status_allocatable_memory_bytes)\n",
+              "expr": "sum(min without(resource) (kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=~\"(memory|requests.memory)\"}))\n  /\nsum(kube_node_status_allocatable{resource=\"memory\", job=\"kube-state-metrics\"})\n  > 1.5\n",
              "for": "5m",
              "labels": {
                "severity": "warning"
              }
            },
            {
-              "alert": "KubeCPUOvercommit",
+              "alert": "KubeQuotaAlmostFull",
              "annotations": {
-                "message": "Cluster has overcommitted CPU resource requests for Namespaces.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit"
+                "description": "Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotaalmostfull",
+                "summary": "Namespace quota is going to be full."
              },
-              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"cpu\"})\n  /\nsum(kube_node_status_allocatable_cpu_cores)\n  > 1.5\n",
-              "for": "5m",
+              "expr": "kube_resourcequota{job=\"kube-state-metrics\", type=\"used\"}\n  / ignoring(instance, job, type)\n(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\"} > 0)\n  > 0.9 < 1\n",
+              "for": "15m",
              "labels": {
-                "severity": "warning"
+                "severity": "info"
              }
            },
            {
-              "alert": "KubeMemOvercommit",
+              "alert": "KubeQuotaFullyUsed",
              "annotations": {
-                "message": "Cluster has overcommitted memory resource requests for Namespaces.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit"
+                "description": "Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotafullyused",
+                "summary": "Namespace quota is fully used."
              },
-              "expr": "sum(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\", resource=\"memory\"})\n  /\nsum(kube_node_status_allocatable_memory_bytes{job=\"node-exporter\"})\n  > 1.5\n",
-              "for": "5m",
+              "expr": "kube_resourcequota{job=\"kube-state-metrics\", type=\"used\"}\n  / ignoring(instance, job, type)\n(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\"} > 0)\n  == 1\n",
+              "for": "15m",
              "labels": {
-                "severity": "warning"
+                "severity": "info"
              }
            },
            {
              "alert": "KubeQuotaExceeded",
              "annotations": {
-                "message": "Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotaexceeded"
+                "description": "Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotaexceeded",
+                "summary": "Namespace quota has exceeded the limits."
              },
-              "expr": "kube_resourcequota{job=\"kube-state-metrics\", type=\"used\"}\n  / ignoring(instance, job, type)\n(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\"} > 0)\n  > 0.90\n",
+              "expr": "kube_resourcequota{job=\"kube-state-metrics\", type=\"used\"}\n  / ignoring(instance, job, type)\n(kube_resourcequota{job=\"kube-state-metrics\", type=\"hard\"} > 0)\n  > 1\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -718,13 +869,14 @@ data:
            {
              "alert": "CPUThrottlingHigh",
              "annotations": {
-                "message": "{{ $value | humanizePercentage }} throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh"
+                "description": "{{ $value | humanizePercentage }} throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh",
+                "summary": "Processes experience elevated CPU throttling."
              },
-              "expr": "sum(increase(container_cpu_cfs_throttled_periods_total{container!=\"\", }[5m])) by (container, pod, namespace)\n  /\nsum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace)\n  > ( 100 / 100 )\n",
+              "expr": "sum(increase(container_cpu_cfs_throttled_periods_total{container!=\"\", }[5m])) by (container, pod, namespace)\n  /\nsum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace)\n  > ( 80 / 100 )\n",
              "for": "15m",
              "labels": {
-                "severity": "warning"
+                "severity": "info"
              }
            }
          ]
@ -733,34 +885,37 @@ data:
          "name": "kubernetes-storage",
          "rules": [
            {
-              "alert": "KubePersistentVolumeUsageCritical",
+              "alert": "KubePersistentVolumeFillingUp",
              "annotations": {
-                "message": "The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeusagecritical"
+                "description": "The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefillingup",
+                "summary": "PersistentVolume is filling up."
              },
-              "expr": "kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n  /\nkubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n  < 0.03\n",
+              "expr": "(\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 0.03\nand\nkubelet_volume_stats_used_bytes{job=\"kubelet\"} > 0\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_access_mode{ access_mode=\"ReadOnlyMany\"} == 1\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_labels{label_excluded_from_alerts=\"true\"} == 1\n",
              "for": "1m",
              "labels": {
                "severity": "critical"
              }
            },
            {
-              "alert": "KubePersistentVolumeFullInFourDays",
+              "alert": "KubePersistentVolumeFillingUp",
              "annotations": {
-                "message": "Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefullinfourdays"
+                "description": "Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefillingup",
+                "summary": "PersistentVolume is filling up."
              },
-              "expr": "(\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 0.15\nand\npredict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\"}[6h], 4 * 24 * 3600) < 0\n",
+              "expr": "(\n  kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n    /\n  kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n) < 0.15\nand\nkubelet_volume_stats_used_bytes{job=\"kubelet\"} > 0\nand\npredict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\"}[6h], 4 * 24 * 3600) < 0\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_access_mode{ access_mode=\"ReadOnlyMany\"} == 1\nunless on(namespace, persistentvolumeclaim)\nkube_persistentvolumeclaim_labels{label_excluded_from_alerts=\"true\"} == 1\n",
              "for": "1h",
              "labels": {
-                "severity": "critical"
+                "severity": "warning"
              }
            },
            {
              "alert": "KubePersistentVolumeErrors",
              "annotations": {
-                "message": "The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeerrors"
+                "description": "The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeerrors",
+                "summary": "PersistentVolume is having issues with provisioning."
              },
              "expr": "kube_persistentvolume_status_phase{phase=~\"Failed|Pending\",job=\"kube-state-metrics\"} > 0\n",
              "for": "5m",
@ -776,10 +931,11 @@ data:
            {
              "alert": "KubeVersionMismatch",
              "annotations": {
-                "message": "There are {{ $value }} different semantic versions of Kubernetes components running.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch"
+                "description": "There are {{ $value }} different semantic versions of Kubernetes components running.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch",
+                "summary": "Different semantic versions of Kubernetes components running."
              },
-              "expr": "count(count by (gitVersion) (label_replace(kubernetes_build_info{job!~\"kube-dns|coredns\"},\"gitVersion\",\"$1\",\"gitVersion\",\"(v[0-9]*.[0-9]*.[0-9]*).*\"))) > 1\n",
+              "expr": "count(count by (git_version) (label_replace(kubernetes_build_info{job!~\"kube-dns|coredns\"},\"git_version\",\"$1\",\"git_version\",\"(v[0-9]*.[0-9]*).*\"))) > 1\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -788,10 +944,11 @@ data:
            {
              "alert": "KubeClientErrors",
              "annotations": {
-                "message": "Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors.'",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors"
+                "description": "Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors.'",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors",
+                "summary": "Kubernetes API server client is experiencing errors."
              },
-              "expr": "(sum(rate(rest_client_requests_total{code=~\"5..\"}[5m])) by (instance, job)\n  /\nsum(rate(rest_client_requests_total[5m])) by (instance, job))\n> 0.01\n",
+              "expr": "(sum(rate(rest_client_requests_total{code=~\"5..\"}[5m])) by (instance, job, namespace)\n  /\nsum(rate(rest_client_requests_total[5m])) by (instance, job, namespace))\n> 0.01\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -800,30 +957,66 @@ data:
          ]
        },
        {
-          "name": "kube-apiserver-error-alerts",
+          "name": "kube-apiserver-slos",
          "rules": [
            {
-              "alert": "ErrorBudgetBurn",
+              "alert": "KubeAPIErrorBudgetBurn",
              "annotations": {
-                "message": "High requests error budget burn for job=apiserver (current value: {{ $value }})",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-errorbudgetburn"
+                "description": "The API server is burning too much error budget.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorbudgetburn",
+                "summary": "The API server is burning too much error budget."
              },
-              "expr": "(\n  status_class_5xx:apiserver_request_total:ratio_rate1h{job=\"apiserver\"} > (14.4*0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate5m{job=\"apiserver\"} > (14.4*0.010000)\n)\nor\n(\n  status_class_5xx:apiserver_request_total:ratio_rate6h{job=\"apiserver\"} > (6*0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate30m{job=\"apiserver\"} > (6*0.010000)\n)\n",
+              "expr": "sum(apiserver_request:burnrate1h) > (14.40 * 0.01000)\nand\nsum(apiserver_request:burnrate5m) > (14.40 * 0.01000)\n",
+              "for": "2m",
              "labels": {
-                "job": "apiserver",
-                "severity": "critical"
+                "long": "1h",
+                "severity": "critical",
+                "short": "5m"
              }
            },
            {
-              "alert": "ErrorBudgetBurn",
+              "alert": "KubeAPIErrorBudgetBurn",
              "annotations": {
-                "message": "High requests error budget burn for job=apiserver (current value: {{ $value }})",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-errorbudgetburn"
+                "description": "The API server is burning too much error budget.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorbudgetburn",
+                "summary": "The API server is burning too much error budget."
              },
-              "expr": "(\n  status_class_5xx:apiserver_request_total:ratio_rate1d{job=\"apiserver\"} > (3*0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate2h{job=\"apiserver\"} > (3*0.010000)\n)\nor\n(\n  status_class_5xx:apiserver_request_total:ratio_rate3d{job=\"apiserver\"} > (0.010000)\n  and\n  status_class_5xx:apiserver_request_total:ratio_rate6h{job=\"apiserver\"} > (0.010000)\n)\n",
+              "expr": "sum(apiserver_request:burnrate6h) > (6.00 * 0.01000)\nand\nsum(apiserver_request:burnrate30m) > (6.00 * 0.01000)\n",
+              "for": "15m",
              "labels": {
-                "job": "apiserver",
-                "severity": "warning"
+                "long": "6h",
+                "severity": "critical",
+                "short": "30m"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorBudgetBurn",
+              "annotations": {
+                "description": "The API server is burning too much error budget.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorbudgetburn",
+                "summary": "The API server is burning too much error budget."
+              },
+              "expr": "sum(apiserver_request:burnrate1d) > (3.00 * 0.01000)\nand\nsum(apiserver_request:burnrate2h) > (3.00 * 0.01000)\n",
+              "for": "1h",
+              "labels": {
+                "long": "1d",
+                "severity": "warning",
+                "short": "2h"
+              }
+            },
+            {
+              "alert": "KubeAPIErrorBudgetBurn",
+              "annotations": {
+                "description": "The API server is burning too much error budget.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorbudgetburn",
+                "summary": "The API server is burning too much error budget."
+              },
+              "expr": "sum(apiserver_request:burnrate3d) > (1.00 * 0.01000)\nand\nsum(apiserver_request:burnrate6h) > (1.00 * 0.01000)\n",
+              "for": "3h",
+              "labels": {
+                "long": "3d",
+                "severity": "warning",
+                "short": "6h"
              }
            }
          ]
@ -831,59 +1024,12 @@ data:
        {
          "name": "kubernetes-system-apiserver",
          "rules": [
-            {
-              "alert": "KubeAPILatencyHigh",
-              "annotations": {
-                "message": "The API server has an abnormal latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh"
-              },
-              "expr": "(\n  cluster:apiserver_request_duration_seconds:mean5m{job=\"apiserver\"}\n  >\n  on (verb) group_left()\n  (\n    avg by (verb) (cluster:apiserver_request_duration_seconds:mean5m{job=\"apiserver\"} >= 0)\n    +\n    2*stddev by (verb) (cluster:apiserver_request_duration_seconds:mean5m{job=\"apiserver\"} >= 0)\n  )\n) > on (verb) group_left()\n1.2 * avg by (verb) (cluster:apiserver_request_duration_seconds:mean5m{job=\"apiserver\"} >= 0)\nand on (verb,resource)\ncluster_quantile:apiserver_request_duration_seconds:histogram_quantile{job=\"apiserver\",quantile=\"0.99\"}\n>\n1\n",
-              "for": "5m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
-            {
-              "alert": "KubeAPILatencyHigh",
-              "annotations": {
-                "message": "The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh"
-              },
-              "expr": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{job=\"apiserver\",quantile=\"0.99\"} > 4\n",
-              "for": "10m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
-            {
-              "alert": "KubeAPIErrorsHigh",
-              "annotations": {
-                "message": "API server is returning errors for {{ $value | humanizePercentage }} of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
-              },
-              "expr": "sum(rate(apiserver_request_total{job=\"apiserver\",code=~\"5..\"}[5m])) by (resource,subresource,verb)\n  /\nsum(rate(apiserver_request_total{job=\"apiserver\"}[5m])) by (resource,subresource,verb) > 0.10\n",
-              "for": "10m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
-            {
-              "alert": "KubeAPIErrorsHigh",
-              "annotations": {
-                "message": "API server is returning errors for {{ $value | humanizePercentage }} of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh"
-              },
-              "expr": "sum(rate(apiserver_request_total{job=\"apiserver\",code=~\"5..\"}[5m])) by (resource,subresource,verb)\n  /\nsum(rate(apiserver_request_total{job=\"apiserver\"}[5m])) by (resource,subresource,verb) > 0.05\n",
-              "for": "10m",
-              "labels": {
-                "severity": "warning"
-              }
-            },
            {
              "alert": "KubeClientCertificateExpiration",
              "annotations": {
-                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 1.0 hours.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
+                "description": "A client certificate used to authenticate to kubernetes apiserver is expiring in less than 1.0 hours.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration",
+                "summary": "Client certificate is about to expire."
              },
              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 3600\n",
              "labels": {
@ -893,8 +1039,9 @@ data:
            {
              "alert": "KubeClientCertificateExpiration",
              "annotations": {
-                "message": "A client certificate used to authenticate to the apiserver is expiring in less than 0.1 hours.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration"
+                "description": "A client certificate used to authenticate to kubernetes apiserver is expiring in less than 0.1 hours.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration",
+                "summary": "Client certificate is about to expire."
              },
              "expr": "apiserver_client_certificate_expiration_seconds_count{job=\"apiserver\"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"apiserver\"}[5m]))) < 300\n",
              "labels": {
@ -902,23 +1049,25 @@ data:
              }
            },
            {
-              "alert": "AggregatedAPIErrors",
+              "alert": "KubeAggregatedAPIErrors",
              "annotations": {
-                "message": "An aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. The number of errors have increased for it in the past five minutes. High values indicate that the availability of the service changes too often.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-aggregatedapierrors"
+                "description": "Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. It has appeared unavailable {{ $value | humanize }} times averaged over the past 10m.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeaggregatedapierrors",
+                "summary": "Kubernetes aggregated API has reported errors."
              },
-              "expr": "sum by(name, namespace)(increase(aggregator_unavailable_apiservice_count[5m])) > 2\n",
+              "expr": "sum by(name, namespace)(increase(aggregator_unavailable_apiservice_total[10m])) > 4\n",
              "labels": {
                "severity": "warning"
              }
            },
            {
-              "alert": "AggregatedAPIDown",
+              "alert": "KubeAggregatedAPIDown",
              "annotations": {
-                "message": "An aggregated API {{ $labels.name }}/{{ $labels.namespace }} is down. It has not been available at least for the past five minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-aggregatedapidown"
+                "description": "Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeaggregatedapidown",
+                "summary": "Kubernetes aggregated API is down."
              },
-              "expr": "sum by(name, namespace)(sum_over_time(aggregator_unavailable_apiservice[5m])) > 0\n",
+              "expr": "(1 - max by(name, namespace)(avg_over_time(aggregator_unavailable_apiservice[10m]))) * 100 < 85\n",
              "for": "5m",
              "labels": {
                "severity": "warning"
@ -927,14 +1076,28 @@ data:
            {
              "alert": "KubeAPIDown",
              "annotations": {
-                "message": "KubeAPI has disappeared from Prometheus target discovery.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapidown"
+                "description": "KubeAPI has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapidown",
+                "summary": "Target disappeared from Prometheus target discovery."
              },
              "expr": "absent(up{job=\"apiserver\"} == 1)\n",
              "for": "15m",
              "labels": {
                "severity": "critical"
              }
+            },
+            {
+              "alert": "KubeAPITerminatedRequests",
+              "annotations": {
+                "description": "The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapiterminatedrequests",
+                "summary": "The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests."
+              },
+              "expr": "sum(rate(apiserver_request_terminations_total{job=\"apiserver\"}[10m]))  / (  sum(rate(apiserver_request_total{job=\"apiserver\"}[10m])) + sum(rate(apiserver_request_terminations_total{job=\"apiserver\"}[10m])) ) > 0.20\n",
+              "for": "5m",
+              "labels": {
+                "severity": "warning"
+              }
            }
          ]
        },
@ -944,8 +1107,9 @@ data:
            {
              "alert": "KubeNodeNotReady",
              "annotations": {
-                "message": "{{ $labels.node }} has been unready for more than 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodenotready"
+                "description": "{{ $labels.node }} has been unready for more than 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodenotready",
+                "summary": "Node is not ready."
              },
              "expr": "kube_node_status_condition{job=\"kube-state-metrics\",condition=\"Ready\",status=\"true\"} == 0\n",
              "for": "15m",
@ -956,11 +1120,12 @@ data:
            {
              "alert": "KubeNodeUnreachable",
              "annotations": {
-                "message": "{{ $labels.node }} is unreachable and some workloads may be rescheduled.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodeunreachable"
+                "description": "{{ $labels.node }} is unreachable and some workloads may be rescheduled.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodeunreachable",
+                "summary": "Node is unreachable."
              },
-              "expr": "kube_node_spec_taint{job=\"kube-state-metrics\",key=\"node.kubernetes.io/unreachable\",effect=\"NoSchedule\"} == 1\n",
-              "for": "2m",
+              "expr": "(kube_node_spec_taint{job=\"kube-state-metrics\",key=\"node.kubernetes.io/unreachable\",effect=\"NoSchedule\"} unless ignoring(key,value) kube_node_spec_taint{job=\"kube-state-metrics\",key=~\"ToBeDeletedByClusterAutoscaler|cloud.google.com/impending-node-termination|aws-node-termination-handler/spot-itn\"}) == 1\n",
+              "for": "15m",
              "labels": {
                "severity": "warning"
              }
@ -968,20 +1133,22 @@ data:
            {
              "alert": "KubeletTooManyPods",
              "annotations": {
-                "message": "Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods"
+                "description": "Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods",
+                "summary": "Kubelet is running at capacity."
              },
-              "expr": "max(max(kubelet_running_pod_count{job=\"kubelet\"}) by(instance) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"}) by(node) / max(kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) by(node) > 0.95\n",
+              "expr": "count by(node) (\n  (kube_pod_status_phase{job=\"kube-state-metrics\",phase=\"Running\"} == 1) * on(instance,pod,namespace,cluster) group_left(node) topk by(instance,pod,namespace,cluster) (1, kube_pod_info{job=\"kube-state-metrics\"})\n)\n/\nmax by(node) (\n  kube_node_status_capacity{job=\"kube-state-metrics\",resource=\"pods\"} != 1\n) > 0.95\n",
              "for": "15m",
              "labels": {
-                "severity": "warning"
+                "severity": "info"
              }
            },
            {
              "alert": "KubeNodeReadinessFlapping",
              "annotations": {
-                "message": "The readiness status of node {{ $labels.node }} has changed {{ $value }} times in the last 15 minutes.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodereadinessflapping"
+                "description": "The readiness status of node {{ $labels.node }} has changed {{ $value }} times in the last 15 minutes.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodereadinessflapping",
+                "summary": "Node readiness status is flapping."
              },
              "expr": "sum(changes(kube_node_status_condition{status=\"true\",condition=\"Ready\"}[15m])) by (node) > 2\n",
              "for": "15m",
@ -992,8 +1159,9 @@ data:
            {
              "alert": "KubeletPlegDurationHigh",
              "annotations": {
-                "message": "The Kubelet Pod Lifecycle Event Generator has a 99th percentile duration of {{ $value }} seconds on node {{ $labels.node }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletplegdurationhigh"
+                "description": "The Kubelet Pod Lifecycle Event Generator has a 99th percentile duration of {{ $value }} seconds on node {{ $labels.node }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletplegdurationhigh",
+                "summary": "Kubelet Pod Lifecycle Event Generator is taking too long to relist."
              },
              "expr": "node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile{quantile=\"0.99\"} >= 10\n",
              "for": "5m",
@ -1004,10 +1172,85 @@ data:
            {
              "alert": "KubeletPodStartUpLatencyHigh",
              "annotations": {
-                "message": "Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }}.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh"
+                "description": "Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh",
+                "summary": "Kubelet Pod startup latency is too high."
              },
-              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\"}[5m])) by (instance, le)) * on(instance) group_left(node) kubelet_node_name  > 60\n",
+              "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\"}[5m])) by (instance, le)) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\"} > 60\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeletClientCertificateExpiration",
+              "annotations": {
+                "description": "Client certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletclientcertificateexpiration",
+                "summary": "Kubelet client certificate is about to expire."
+              },
+              "expr": "kubelet_certificate_manager_client_ttl_seconds < 3600\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeletClientCertificateExpiration",
+              "annotations": {
+                "description": "Client certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletclientcertificateexpiration",
+                "summary": "Kubelet client certificate is about to expire."
+              },
+              "expr": "kubelet_certificate_manager_client_ttl_seconds < 300\n",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeletServerCertificateExpiration",
+              "annotations": {
+                "description": "Server certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletservercertificateexpiration",
+                "summary": "Kubelet server certificate is about to expire."
+              },
+              "expr": "kubelet_certificate_manager_server_ttl_seconds < 3600\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeletServerCertificateExpiration",
+              "annotations": {
+                "description": "Server certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }}.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletservercertificateexpiration",
+                "summary": "Kubelet server certificate is about to expire."
+              },
+              "expr": "kubelet_certificate_manager_server_ttl_seconds < 300\n",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "KubeletClientCertificateRenewalErrors",
+              "annotations": {
+                "description": "Kubelet on node {{ $labels.node }} has failed to renew its client certificate ({{ $value | humanize }} errors in the last 5 minutes).",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletclientcertificaterenewalerrors",
+                "summary": "Kubelet has failed to renew its client certificate."
+              },
+              "expr": "increase(kubelet_certificate_manager_client_expiration_renew_errors[5m]) > 0\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "KubeletServerCertificateRenewalErrors",
+              "annotations": {
+                "description": "Kubelet on node {{ $labels.node }} has failed to renew its server certificate ({{ $value | humanize }} errors in the last 5 minutes).",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletservercertificaterenewalerrors",
+                "summary": "Kubelet has failed to renew its server certificate."
+              },
+              "expr": "increase(kubelet_server_expiration_renew_errors[5m]) > 0\n",
              "for": "15m",
              "labels": {
                "severity": "warning"
@ -1016,8 +1259,9 @@ data:
            {
              "alert": "KubeletDown",
              "annotations": {
-                "message": "Kubelet has disappeared from Prometheus target discovery.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown"
+                "description": "Kubelet has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown",
+                "summary": "Target disappeared from Prometheus target discovery."
              },
              "expr": "absent(up{job=\"kubelet\"} == 1)\n",
              "for": "15m",
@ -1033,8 +1277,9 @@ data:
            {
              "alert": "KubeSchedulerDown",
              "annotations": {
-                "message": "KubeScheduler has disappeared from Prometheus target discovery.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeschedulerdown"
+                "description": "KubeScheduler has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeschedulerdown",
+                "summary": "Target disappeared from Prometheus target discovery."
              },
              "expr": "absent(up{job=\"kube-scheduler\"} == 1)\n",
              "for": "15m",
@ -1050,8 +1295,9 @@ data:
            {
              "alert": "KubeControllerManagerDown",
              "annotations": {
-                "message": "KubeControllerManager has disappeared from Prometheus target discovery.",
-                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown"
+                "description": "KubeControllerManager has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown",
+                "summary": "Target disappeared from Prometheus target discovery."
              },
              "expr": "absent(up{job=\"kube-controller-manager\"} == 1)\n",
              "for": "15m",
@ -1060,109 +1306,18 @@ data:
              }
            }
          ]
-        }
-      ]
-    }
-  loki.yaml: |-
-    {
-      "groups": [
-        {
-          "name": "loki_rules",
-          "rules": [
-            {
-              "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job))",
-              "record": "job:loki_request_duration_seconds:99quantile"
-            },
-            {
-              "expr": "histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job))",
-              "record": "job:loki_request_duration_seconds:50quantile"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job) / sum(rate(loki_request_duration_seconds_count[1m])) by (job)",
-              "record": "job:loki_request_duration_seconds:avg"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job)",
-              "record": "job:loki_request_duration_seconds_bucket:sum_rate"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job)",
-              "record": "job:loki_request_duration_seconds_sum:sum_rate"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_count[1m])) by (job)",
-              "record": "job:loki_request_duration_seconds_count:sum_rate"
-            },
-            {
-              "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route))",
-              "record": "job_route:loki_request_duration_seconds:99quantile"
-            },
-            {
-              "expr": "histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route))",
-              "record": "job_route:loki_request_duration_seconds:50quantile"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job, route) / sum(rate(loki_request_duration_seconds_count[1m])) by (job, route)",
-              "record": "job_route:loki_request_duration_seconds:avg"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route)",
-              "record": "job_route:loki_request_duration_seconds_bucket:sum_rate"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (job, route)",
-              "record": "job_route:loki_request_duration_seconds_sum:sum_rate"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_count[1m])) by (job, route)",
-              "record": "job_route:loki_request_duration_seconds_count:sum_rate"
-            },
-            {
-              "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, namespace, job, route))",
-              "record": "namespace_job_route:loki_request_duration_seconds:99quantile"
-            },
-            {
-              "expr": "histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, namespace, job, route))",
-              "record": "namespace_job_route:loki_request_duration_seconds:50quantile"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (namespace, job, route) / sum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route)",
-              "record": "namespace_job_route:loki_request_duration_seconds:avg"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, namespace, job, route)",
-              "record": "namespace_job_route:loki_request_duration_seconds_bucket:sum_rate"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_sum[1m])) by (namespace, job, route)",
-              "record": "namespace_job_route:loki_request_duration_seconds_sum:sum_rate"
-            },
-            {
-              "expr": "sum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route)",
-              "record": "namespace_job_route:loki_request_duration_seconds_count:sum_rate"
-            }
-          ]
        },
        {
-          "name": "loki_alerts",
+          "name": "kubernetes-system-kube-proxy",
          "rules": [
            {
-              "alert": "LokiRequestErrors",
+              "alert": "KubeProxyDown",
              "annotations": {
-                "message": "{{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}% errors.\n"
+                "description": "KubeProxy has disappeared from Prometheus target discovery.",
+                "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeproxydown",
+                "summary": "Target disappeared from Prometheus target discovery."
              },
-              "expr": "100 * sum(rate(loki_request_duration_seconds_count{status_code=~\"5..\"}[1m])) by (namespace, job, route)\n  /\nsum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route)\n  > 10\n",
-              "for": "15m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
-            {
-              "alert": "LokiRequestLatency",
-              "annotations": {
-                "message": "{{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency.\n"
-              },
-              "expr": "namespace_job_route:loki_request_duration_seconds:99quantile{route!~\"(?i).*tail.*\"} > 1\n",
+              "expr": "absent(up{job=\"kube-proxy\"} == 1)\n",
              "for": "15m",
              "labels": {
                "severity": "critical"
@ -1179,48 +1334,48 @@ data:
          "name": "node-exporter.rules",
          "rules": [
            {
-              "expr": "count without (cpu) (\n  count without (mode) (\n    node_cpu_seconds_total{job=\"node-exporter\"}\n  )\n)\n",
+              "expr": "count without (cpu, mode) (\n  node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}\n)\n",
              "record": "instance:node_num_cpu:sum"
            },
            {
-              "expr": "1 - avg without (cpu, mode) (\n  rate(node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\"}[1m])\n)\n",
-              "record": "instance:node_cpu_utilisation:rate1m"
+              "expr": "1 - avg without (cpu) (\n  sum without (mode) (rate(node_cpu_seconds_total{job=\"node-exporter\", mode=~\"idle|iowait|steal\"}[5m]))\n)\n",
+              "record": "instance:node_cpu_utilisation:rate5m"
            },
            {
              "expr": "(\n  node_load1{job=\"node-exporter\"}\n/\n  instance:node_num_cpu:sum{job=\"node-exporter\"}\n)\n",
              "record": "instance:node_load1_per_cpu:ratio"
            },
            {
-              "expr": "1 - (\n  node_memory_MemAvailable_bytes{job=\"node-exporter\"}\n/\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n)\n",
+              "expr": "1 - (\n  (\n    node_memory_MemAvailable_bytes{job=\"node-exporter\"}\n    or\n    (\n      node_memory_Buffers_bytes{job=\"node-exporter\"}\n      +\n      node_memory_Cached_bytes{job=\"node-exporter\"}\n      +\n      node_memory_MemFree_bytes{job=\"node-exporter\"}\n      +\n      node_memory_Slab_bytes{job=\"node-exporter\"}\n    )\n  )\n/\n  node_memory_MemTotal_bytes{job=\"node-exporter\"}\n)\n",
              "record": "instance:node_memory_utilisation:ratio"
            },
            {
-              "expr": "rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[1m])\n",
-              "record": "instance:node_vmstat_pgmajfault:rate1m"
+              "expr": "rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[5m])\n",
+              "record": "instance:node_vmstat_pgmajfault:rate5m"
            },
            {
-              "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[1m])\n",
-              "record": "instance_device:node_disk_io_time_seconds:rate1m"
+              "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[5m])\n",
+              "record": "instance_device:node_disk_io_time_seconds:rate5m"
            },
            {
-              "expr": "rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[1m])\n",
-              "record": "instance_device:node_disk_io_time_weighted_seconds:rate1m"
+              "expr": "rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device!~\"dm.*\"}[5m])\n",
+              "record": "instance_device:node_disk_io_time_weighted_seconds:rate5m"
            },
            {
-              "expr": "sum without (device) (\n  rate(node_network_receive_bytes_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
-              "record": "instance:node_network_receive_bytes_excluding_lo:rate1m"
+              "expr": "sum without (device) (\n  rate(node_network_receive_bytes_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
+              "record": "instance:node_network_receive_bytes_excluding_lo:rate5m"
            },
            {
-              "expr": "sum without (device) (\n  rate(node_network_transmit_bytes_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
-              "record": "instance:node_network_transmit_bytes_excluding_lo:rate1m"
+              "expr": "sum without (device) (\n  rate(node_network_transmit_bytes_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
+              "record": "instance:node_network_transmit_bytes_excluding_lo:rate5m"
            },
            {
-              "expr": "sum without (device) (\n  rate(node_network_receive_drop_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
-              "record": "instance:node_network_receive_drop_excluding_lo:rate1m"
+              "expr": "sum without (device) (\n  rate(node_network_receive_drop_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
+              "record": "instance:node_network_receive_drop_excluding_lo:rate5m"
            },
            {
-              "expr": "sum without (device) (\n  rate(node_network_transmit_drop_total{job=\"node-exporter\", device!=\"lo\"}[1m])\n)\n",
-              "record": "instance:node_network_transmit_drop_excluding_lo:rate1m"
+              "expr": "sum without (device) (\n  rate(node_network_transmit_drop_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n",
+              "record": "instance:node_network_transmit_drop_excluding_lo:rate5m"
            }
          ]
        },
@ -1258,7 +1413,7 @@ data:
                "summary": "Filesystem has less than 5% space left."
              },
              "expr": "(\n  node_filesystem_avail_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} * 100 < 5\nand\n  node_filesystem_readonly{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} == 0\n)\n",
-              "for": "1h",
+              "for": "30m",
              "labels": {
                "severity": "warning"
              }
@ -1270,7 +1425,7 @@ data:
                "summary": "Filesystem has less than 3% space left."
              },
              "expr": "(\n  node_filesystem_avail_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} * 100 < 3\nand\n  node_filesystem_readonly{job=\"node-exporter\",fstype!~\"tmpfs|nsfs|vfat\"} == 0\n)\n",
-              "for": "1h",
+              "for": "30m",
              "labels": {
                "severity": "critical"
              }
@ -1329,7 +1484,7 @@ data:
                "description": "{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \"%.0f\" $value }} receive errors in the last two minutes.",
                "summary": "Network interface is reporting many receive errors."
              },
-              "expr": "increase(node_network_receive_errs_total[2m]) > 10\n",
+              "expr": "rate(node_network_receive_errs_total[2m]) / rate(node_network_receive_packets_total[2m]) > 0.01\n",
              "for": "1h",
              "labels": {
                "severity": "warning"
@ -1341,7 +1496,7 @@ data:
                "description": "{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \"%.0f\" $value }} transmit errors in the last two minutes.",
                "summary": "Network interface is reporting many transmit errors."
              },
-              "expr": "increase(node_network_transmit_errs_total[2m]) > 10\n",
+              "expr": "rate(node_network_transmit_errs_total[2m]) / rate(node_network_transmit_packets_total[2m]) > 0.01\n",
              "for": "1h",
              "labels": {
                "severity": "warning"
@ -1350,18 +1505,29 @@ data:
            {
              "alert": "NodeHighNumberConntrackEntriesUsed",
              "annotations": {
-                "description": "{{ $value | humanizePercentage }} of conntrack entries are used",
-                "summary": "Number of conntrack are getting close to the limit"
+                "description": "{{ $value | humanizePercentage }} of conntrack entries are used.",
+                "summary": "Number of conntrack are getting close to the limit."
              },
              "expr": "(node_nf_conntrack_entries / node_nf_conntrack_entries_limit) > 0.75\n",
              "labels": {
                "severity": "warning"
              }
            },
+            {
+              "alert": "NodeTextFileCollectorScrapeError",
+              "annotations": {
+                "description": "Node Exporter text file collector failed to scrape.",
+                "summary": "Node Exporter text file collector failed to scrape."
+              },
+              "expr": "node_textfile_scrape_error{job=\"node-exporter\"} == 1\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
            {
              "alert": "NodeClockSkewDetected",
              "annotations": {
-                "message": "Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.",
+                "description": "Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.",
                "summary": "Clock skew detected."
              },
              "expr": "(\n  node_timex_offset_seconds > 0.05\nand\n  deriv(node_timex_offset_seconds[5m]) >= 0\n)\nor\n(\n  node_timex_offset_seconds < -0.05\nand\n  deriv(node_timex_offset_seconds[5m]) <= 0\n)\n",
@ -1373,14 +1539,61 @@ data:
            {
              "alert": "NodeClockNotSynchronising",
              "annotations": {
-                "message": "Clock on {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.",
+                "description": "Clock on {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.",
                "summary": "Clock not synchronising."
              },
-              "expr": "min_over_time(node_timex_sync_status[5m]) == 0\n",
+              "expr": "min_over_time(node_timex_sync_status[5m]) == 0\nand\nnode_timex_maxerror_seconds >= 16\n",
              "for": "10m",
              "labels": {
                "severity": "warning"
              }
+            },
+            {
+              "alert": "NodeRAIDDegraded",
+              "annotations": {
+                "description": "RAID array '{{ $labels.device }}' on {{ $labels.instance }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.",
+                "summary": "RAID Array is degraded"
+              },
+              "expr": "node_md_disks_required - ignoring (state) (node_md_disks{state=\"active\"}) > 0\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "NodeRAIDDiskFailure",
+              "annotations": {
+                "description": "At least one device in RAID array on {{ $labels.instance }} failed. Array '{{ $labels.device }}' needs attention and possibly a disk swap.",
+                "summary": "Failed device in RAID array"
+              },
+              "expr": "node_md_disks{state=\"failed\"} > 0\n",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeFileDescriptorLimit",
+              "annotations": {
+                "description": "File descriptors limit at {{ $labels.instance }} is currently at {{ printf \"%.2f\" $value }}%.",
+                "summary": "Kernel is predicted to exhaust file descriptors limit soon."
+              },
+              "expr": "(\n  node_filefd_allocated{job=\"node-exporter\"} * 100 / node_filefd_maximum{job=\"node-exporter\"} > 70\n)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "NodeFileDescriptorLimit",
+              "annotations": {
+                "description": "File descriptors limit at {{ $labels.instance }} is currently at {{ printf \"%.2f\" $value }}%.",
+                "summary": "Kernel is predicted to exhaust file descriptors limit soon."
+              },
+              "expr": "(\n  node_filefd_allocated{job=\"node-exporter\"} * 100 / node_filefd_maximum{job=\"node-exporter\"} > 90\n)\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
            }
          ]
        }
@ -1428,18 +1641,6 @@ data:
                "severity": "warning"
              }
            },
-            {
-              "alert": "PrometheusErrorSendingAlertsToAnyAlertmanager",
-              "annotations": {
-                "description": "{{ printf \"%.1f\" $value }}% minimum errors while sending alerts from Prometheus {{$labels.instance}} to any Alertmanager.",
-                "summary": "Prometheus encounters more than 3% errors sending alerts to any Alertmanager."
-              },
-              "expr": "min without(alertmanager) (\n  rate(prometheus_notifications_errors_total{job=\"prometheus\"}[5m])\n/\n  rate(prometheus_notifications_sent_total{job=\"prometheus\"}[5m])\n)\n* 100\n> 3\n",
-              "for": "15m",
-              "labels": {
-                "severity": "critical"
-              }
-            },
            {
              "alert": "PrometheusNotConnectedToAlertmanagers",
              "annotations": {
@ -1482,7 +1683,7 @@ data:
                "description": "Prometheus {{$labels.instance}} is not ingesting samples.",
                "summary": "Prometheus is not ingesting samples."
              },
-              "expr": "rate(prometheus_tsdb_head_samples_appended_total{job=\"prometheus\"}[5m]) <= 0\n",
+              "expr": "(\n  rate(prometheus_tsdb_head_samples_appended_total{job=\"prometheus\"}[5m]) <= 0\nand\n  (\n    sum without(scrape_job) (prometheus_target_metadata_cache_entries{job=\"prometheus\"}) > 0\n  or\n    sum without(rule_group) (prometheus_rule_group_rules{job=\"prometheus\"}) > 0\n  )\n)\n",
              "for": "10m",
              "labels": {
                "severity": "warning"
@ -1515,10 +1716,10 @@ data:
            {
              "alert": "PrometheusRemoteStorageFailures",
              "annotations": {
-                "description": "Prometheus {{$labels.instance}} failed to send {{ printf \"%.1f\" $value }}% of the samples to {{ if $labels.queue }}{{ $labels.queue }}{{ else }}{{ $labels.url }}{{ end }}.",
+                "description": "Prometheus {{$labels.instance}} failed to send {{ printf \"%.1f\" $value }}% of the samples to {{ $labels.remote_name}}:{{ $labels.url }}",
                "summary": "Prometheus fails to send samples to remote storage."
              },
-              "expr": "(\n  rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m])\n/\n  (\n    rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m])\n  +\n    rate(prometheus_remote_storage_succeeded_samples_total{job=\"prometheus\"}[5m])\n  )\n)\n* 100\n> 1\n",
+              "expr": "(\n  (rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{job=\"prometheus\"}[5m]))\n/\n  (\n    (rate(prometheus_remote_storage_failed_samples_total{job=\"prometheus\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{job=\"prometheus\"}[5m]))\n  +\n    (rate(prometheus_remote_storage_succeeded_samples_total{job=\"prometheus\"}[5m]) or rate(prometheus_remote_storage_samples_total{job=\"prometheus\"}[5m]))\n  )\n)\n* 100\n> 1\n",
              "for": "15m",
              "labels": {
                "severity": "critical"
@ -1527,10 +1728,10 @@ data:
            {
              "alert": "PrometheusRemoteWriteBehind",
              "annotations": {
-                "description": "Prometheus {{$labels.instance}} remote write is {{ printf \"%.1f\" $value }}s behind for {{ if $labels.queue }}{{ $labels.queue }}{{ else }}{{ $labels.url }}{{ end }}.",
+                "description": "Prometheus {{$labels.instance}} remote write is {{ printf \"%.1f\" $value }}s behind for {{ $labels.remote_name}}:{{ $labels.url }}.",
                "summary": "Prometheus remote write is behind."
              },
-              "expr": "# Without max_over_time, failed scrapes could create false negatives, see\n# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.\n(\n  max_over_time(prometheus_remote_storage_highest_timestamp_in_seconds{job=\"prometheus\"}[5m])\n- on(job, instance) group_right\n  max_over_time(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{job=\"prometheus\"}[5m])\n)\n> 120\n",
+              "expr": "# Without max_over_time, failed scrapes could create false negatives, see\n# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.\n(\n  max_over_time(prometheus_remote_storage_highest_timestamp_in_seconds{job=\"prometheus\"}[5m])\n- ignoring(remote_name, url) group_right\n  max_over_time(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{job=\"prometheus\"}[5m])\n)\n> 120\n",
              "for": "15m",
              "labels": {
                "severity": "critical"
@ -1539,7 +1740,7 @@ data:
            {
              "alert": "PrometheusRemoteWriteDesiredShards",
              "annotations": {
-                "description": "Prometheus {{$labels.instance}} remote write desired shards calculation wants to run {{ $value }} shards, which is more than the max of {{ printf `prometheus_remote_storage_shards_max{instance=\"%s\",job=\"prometheus\"}` $labels.instance | query | first | value }}.",
+                "description": "Prometheus {{$labels.instance}} remote write desired shards calculation wants to run {{ $value }} shards for queue {{ $labels.remote_name}}:{{ $labels.url }}, which is more than the max of {{ printf `prometheus_remote_storage_shards_max{instance=\"%s\",job=\"prometheus\"}` $labels.instance | query | first | value }}.",
                "summary": "Prometheus remote write desired shards calculation wants to run more than configured max shards."
              },
              "expr": "# Without max_over_time, failed scrapes could create false negatives, see\n# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.\n(\n  max_over_time(prometheus_remote_storage_shards_desired{job=\"prometheus\"}[5m])\n>\n  max_over_time(prometheus_remote_storage_shards_max{job=\"prometheus\"}[5m])\n)\n",
@ -1571,6 +1772,54 @@ data:
              "labels": {
                "severity": "warning"
              }
+            },
+            {
+              "alert": "PrometheusTargetLimitHit",
+              "annotations": {
+                "description": "Prometheus {{$labels.instance}} has dropped {{ printf \"%.0f\" $value }} targets because the number of targets exceeded the configured target_limit.",
+                "summary": "Prometheus has dropped targets because some scrape configs have exceeded the targets limit."
+              },
+              "expr": "increase(prometheus_target_scrape_pool_exceeded_target_limit_total{job=\"prometheus\"}[5m]) > 0\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusLabelLimitHit",
+              "annotations": {
+                "description": "Prometheus {{$labels.instance}} has dropped {{ printf \"%.0f\" $value }} targets because some samples exceeded the configured label_limit, label_name_length_limit or label_value_length_limit.",
+                "summary": "Prometheus has dropped targets because some scrape configs have exceeded the labels limit."
+              },
+              "expr": "increase(prometheus_target_scrape_pool_exceeded_label_limits_total{job=\"prometheus\"}[5m]) > 0\n",
+              "for": "15m",
+              "labels": {
+                "severity": "warning"
+              }
+            },
+            {
+              "alert": "PrometheusTargetSyncFailure",
+              "annotations": {
+                "description": "{{ printf \"%.0f\" $value }} targets in Prometheus {{$labels.instance}} have failed to sync because invalid configuration was supplied.",
+                "summary": "Prometheus has failed to sync targets."
+              },
+              "expr": "increase(prometheus_target_sync_failed_total{job=\"prometheus\"}[30m]) > 0\n",
+              "for": "5m",
+              "labels": {
+                "severity": "critical"
+              }
+            },
+            {
+              "alert": "PrometheusErrorSendingAlertsToAnyAlertmanager",
+              "annotations": {
+                "description": "{{ printf \"%.1f\" $value }}% minimum errors while sending alerts from Prometheus {{$labels.instance}} to any Alertmanager.",
+                "summary": "Prometheus encounters more than 3% errors sending alerts to any Alertmanager."
+              },
+              "expr": "min without (alertmanager) (\n  rate(prometheus_notifications_errors_total{job=\"prometheus\",alertmanager!~``}[5m])\n/\n  rate(prometheus_notifications_sent_total{job=\"prometheus\",alertmanager!~``}[5m])\n)\n* 100\n> 3\n",
+              "for": "15m",
+              "labels": {
+                "severity": "critical"
+              }
            }
          ]
        }
--- a/aws/container-linux/kubernetes/cl/controller.yaml
+++ b/aws/container-linux/kubernetes/cl/controller.yaml
@ -1,199 +0,0 @@
---
-systemd:
-  units:
-    - name: etcd-member.service
-      enabled: true
-      dropins:
-        - name: 40-etcd-cluster.conf
-          contents: |
-            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.9"
-            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
-            Environment="RKT_RUN_ARGS=--insecure-options=image"
-            Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
-            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
-            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
-            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
-            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
-            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
-            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
-            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
-            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
-            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
-            Environment="ETCD_CLIENT_CERT_AUTH=true"
-            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
-            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
-            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
-            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-        RequiredBy=etcd-member.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.6
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=10
-        [Install]
-        WantedBy=multi-user.target
-    - name: bootstrap.service
-      contents: |
-        [Unit]
-        Description=Kubernetes control plane
-        ConditionPathExists=!/opt/bootstrap/bootstrap.done
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        WorkingDirectory=/opt/bootstrap
-        ExecStart=/usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes/bootstrap-secrets \
-            --mount volume=config,target=/etc/kubernetes/secrets \
-            --volume assets,kind=host,source=/opt/bootstrap/assets \
-            --mount volume=assets,target=/assets \
-            --volume script,kind=host,source=/opt/bootstrap/apply \
-            --mount volume=script,target=/apply \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.6 \
-            --net=host \
-            --dns=host \
-            --exec=/apply
-        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /opt/bootstrap/layout
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
-          awk '/#####/ {filename=$2; next} {print > filename}' assets
-          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
-          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
-          chown -R etcd:etcd /etc/ssl/etcd
-          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          mkdir -p /etc/kubernetes/manifests
-          mv static-manifests/* /etc/kubernetes/manifests/
-          mkdir -p /opt/bootstrap/assets
-          mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls manifests-networking
-    - path: /opt/bootstrap/apply
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
-          until kubectl version; do
-            echo "Waiting for static pod control plane"
-            sleep 5
-          done
-          until kubectl apply -f /assets/manifests -R; do
-             echo "Retry applying manifests"
-             sleep 5
-          done
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/versions.tf
+++ b/aws/container-linux/kubernetes/versions.tf
@ -1,11 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = "~> 0.12.6"
-  required_providers {
-    aws      = "~> 2.23"
-    ct       = "~> 0.4"
-    template = "~> 2.1"
-    null     = "~> 2.1"
-  }
-}
--- a/aws/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/aws/container-linux/kubernetes/workers/cl/worker.yaml
@ -1,140 +0,0 @@
---
-systemd:
-  units:
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.6
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/node \
-          %{~ for label in split(",", node_labels) ~}
-          --node-labels=${label} \
-          %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=5
-        [Install]
-        WantedBy=multi-user.target
-    - name: delete-node.service
-      enable: true
-      contents: |
-        [Unit]
-        Description=Waiting to delete Kubernetes node on shutdown
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/etc/kubernetes/delete-node
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-    - path: /etc/kubernetes/delete-node
-      filesystem: root
-      mode: 0744
-      contents:
-        inline: |
-          #!/bin/bash
-          set -e
-          exec /usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes \
-            --mount volume=config,target=/etc/kubernetes \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.6 \
-            --net=host \
-            --dns=host \
-            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname)
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/workers/versions.tf
+++ b/aws/container-linux/kubernetes/workers/versions.tf
@ -1,4 +0,0 @@
-
-terraform {
-  required_version = ">= 0.12"
-}
--- a/aws/fedora-coreos/kubernetes/README.md
+++ b/aws/fedora-coreos/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.6 (upstream)
+* Kubernetes v1.26.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/fedora-coreos/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/aws/fedora-coreos/kubernetes/ami.tf
+++ b/aws/fedora-coreos/kubernetes/ami.tf
@ -1,4 +1,3 @@
-
 data "aws_ami" "fedora-coreos" {
  most_recent = true
  owners      = ["125523088429"]
@ -18,3 +17,25 @@ data "aws_ami" "fedora-coreos" {
    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
+
+data "aws_ami" "fedora-coreos-arm" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["125523088429"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
+  }
+}
--- a/aws/fedora-coreos/kubernetes/bootstrap.tf
+++ b/aws/fedora-coreos/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=835890025bfb3499aa0cd5a9704e9e7d3e7e5fe5"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4621c6b256129d1ede32ae1f72bcc1473664cb33"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers          = aws_route53_record.etcds.*.fqdn
-  asset_dir             = var.asset_dir
  networking            = var.networking
  network_mtu           = var.network_mtu
  pod_cidr              = var.pod_cidr
@ -13,7 +12,6 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
-
-  trusted_certs_dir = "/etc/pki/tls/certs"
+  daemonset_tolerations = var.daemonset_tolerations
 }

--- a/aws/fedora-coreos/kubernetes/butane/controller.yaml
+++ b/aws/fedora-coreos/kubernetes/butane/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.0.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -8,32 +8,32 @@ systemd:
      contents: |
        [Unit]
        Description=etcd (System Container)
-        Documentation=https://github.com/coreos/etcd
-        Wants=network-online.target network.target
+        Documentation=https://github.com/etcd-io/etcd
+        Wants=network-online.target
        After=network-online.target
        [Service]
-        # https://github.com/opencontainers/runc/pull/1807
-        # Type=notify
-        # NotifyAccess=exec
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.7
        Type=exec
+        ExecStartPre=/bin/mkdir -p /var/lib/etcd
+        ExecStartPre=-/usr/bin/podman rm etcd
+        ExecStart=/usr/bin/podman run --name etcd \
+          --env-file /etc/etcd/etcd.env \
+          --log-driver k8s-file \
+          --network host \
+          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
+          $${ETCD_IMAGE}
+        ExecStop=/usr/bin/podman stop etcd
        Restart=on-failure
        RestartSec=10s
        TimeoutStartSec=0
        LimitNOFILE=40000
-        ExecStartPre=/bin/mkdir -p /var/lib/etcd
-        ExecStartPre=-/usr/bin/podman rm etcd
-        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
-        ExecStart=/usr/bin/podman run --name etcd \
-          --env-file /etc/etcd/etcd.env \
-          --network host \
-          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
-          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.9
-        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -53,10 +53,13 @@ systemd:
      contents: |
        [Unit]
        Description=Kubelet (System Container)
+        Requires=afterburn.service
+        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.6
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        EnvironmentFile=/run/metadata/afterburn
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -64,44 +67,33 @@ systemd:
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/podman rm kubelet
        ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /etc/machine-id:/etc/machine-id:ro \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
          $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
-          --cgroups-per-qos=true \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
+          --config=/etc/kubernetes/kubelet.yaml \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID} \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
        Restart=always
@ -120,15 +112,17 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
+            --volume /etc/kubernetes/pki:/etc/kubernetes/pki:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.6
+            quay.io/poseidon/kubelet:v1.26.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
  directories:
+    - path: /var/lib/etcd
+      mode: 0700
    - path: /etc/kubernetes
    - path: /opt/bootstrap
  files:
@ -137,6 +131,33 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: ${cluster_domain_suffix}
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
    - path: /opt/bootstrap/layout
      mode: 0544
      contents:
@ -145,25 +166,26 @@ storage:
          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
          awk '/#####/ {filename=$2; next} {print > filename}' assets
          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mkdir -p /etc/kubernetes/pki
          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
          chown -R etcd:etcd /etc/ssl/etcd
          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
          mkdir -p /etc/kubernetes/manifests
          mv static-manifests/* /etc/kubernetes/manifests/
          mkdir -p /opt/bootstrap/assets
          mv manifests /opt/bootstrap/assets/manifests
          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
+          chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
        inline: |
          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
          until kubectl version; do
            echo "Waiting for static pod control plane"
            sleep 5
@ -172,6 +194,11 @@ storage:
             echo "Retry applying manifests"
             sleep 5
          done
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -181,6 +208,13 @@ storage:
        inline: |
          net.ipv4.conf.default.rp_filter=0
          net.ipv4.conf.*.rp_filter=0
+    - path: /etc/systemd/network/50-flannel.link
+      contents:
+        inline: |
+          [Match]
+          OriginalName=flannel*
+          [Link]
+          MACAddressPolicy=none
    - path: /etc/systemd/system.conf.d/accounting.conf
      contents:
        inline: |
@ -192,8 +226,6 @@ storage:
      mode: 0644
      contents:
        inline: |
-          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
-          NOTIFY_SOCKET=/run/systemd/notify
          ETCD_NAME=${etcd_name}
          ETCD_DATA_DIR=/var/lib/etcd
          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
@ -211,6 +243,25 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
--- a/aws/fedora-coreos/kubernetes/controllers.tf
+++ b/aws/fedora-coreos/kubernetes/controllers.tf
@ -22,9 +22,8 @@ resource "aws_instance" "controllers" {
  }

  instance_type = var.controller_type
-
-  ami       = data.aws_ami.fedora-coreos.image_id
-  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]
+  ami           = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+  user_data     = data.ct_config.controllers.*.rendered[count.index]

  # storage
  root_block_device {
@ -32,6 +31,7 @@ resource "aws_instance" "controllers" {
    volume_size = var.disk_size
    iops        = var.disk_iops
    encrypted   = true
+    tags        = {}
  }

  # network
@ -47,41 +47,22 @@ resource "aws_instance" "controllers" {
  }
 }

-# Controller Ignition configs
-data "ct_config" "controller-ignitions" {
-  count    = var.controller_count
-  content  = data.template_file.controller-configs.*.rendered[count.index]
-  strict   = true
-  snippets = var.controller_snippets
-}
-
-# Controller Fedora CoreOS configs
-data "template_file" "controller-configs" {
+# Fedora CoreOS controllers
+data "ct_config" "controllers" {
  count = var.controller_count
-
-  template = file("${path.module}/fcc/controller.yaml")
-
-  vars = {
+  content = templatefile("${path.module}/butane/controller.yaml", {
    # Cannot use cyclic dependencies on controllers or their DNS records
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    etcd_initial_cluster = join(",", [
+      for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
+    ])
    kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
    ssh_authorized_key     = var.ssh_authorized_key
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
-  }
+  })
+  strict   = true
+  snippets = var.controller_snippets
 }
-
-data "template_file" "etcds" {
-  count    = var.controller_count
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = count.index
-    cluster_name = var.cluster_name
-    dns_zone     = var.dns_zone
-  }
-}
-
--- a/aws/fedora-coreos/kubernetes/nlb.tf
+++ b/aws/fedora-coreos/kubernetes/nlb.tf
@ -17,6 +17,7 @@ resource "aws_route53_record" "apiserver" {
 resource "aws_lb" "nlb" {
  name               = "${var.cluster_name}-nlb"
  load_balancer_type = "network"
+  ip_address_type    = "dualstack"
  internal           = false

  subnets = aws_subnet.public.*.id
--- a/aws/fedora-coreos/kubernetes/outputs.tf
+++ b/aws/fedora-coreos/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "worker_security_groups" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom load balancing
@ -52,3 +54,10 @@ output "worker_target_group_https" {
  value       = module.workers.target_group_https
 }

+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/aws/fedora-coreos/kubernetes/security.tf
+++ b/aws/fedora-coreos/kubernetes/security.tf
@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
  source_security_group_id = aws_security_group.worker.id
 }

@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
  source_security_group_id = aws_security_group.worker.id
 }

--- a/aws/fedora-coreos/kubernetes/ssh.tf
+++ b/aws/fedora-coreos/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/aws/fedora-coreos/kubernetes/variables.tf
+++ b/aws/fedora-coreos/kubernetes/variables.tf
@ -43,26 +43,31 @@ variable "worker_type" {

 variable "os_stream" {
  type        = string
-  description = "Fedora CoreOs image stream for instances (e.g. stable, testing, next)"
+  description = "Fedora CoreOS image stream for instances (e.g. stable, testing, next)"
  default     = "stable"
+
+  validation {
+    condition     = contains(["stable", "testing", "next"], var.os_stream)
+    error_message = "The os_stream must be stable, testing, or next."
+  }
 }

 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -79,13 +84,13 @@ variable "worker_target_groups" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -96,16 +101,10 @@ variable "ssh_authorized_key" {
  description = "SSH public key for user 'core'"
 }

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "networking" {
  type        = string
-  description = "Choice of networking provider (calico or flannel)"
-  default     = "calico"
+  description = "Choice of networking provider (flannel, calico, or cilium)"
+  default     = "cilium"
 }

 variable "network_mtu" {
@ -143,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -161,3 +160,19 @@ variable "cluster_domain_suffix" {
  default     = "cluster.local"
 }

+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
+
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
--- a/aws/fedora-coreos/kubernetes/versions.tf
+++ b/aws/fedora-coreos/kubernetes/versions.tf
@ -1,11 +1,13 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = "~> 0.12.6"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    aws      = "~> 2.23"
-    ct       = "~> 0.4"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    aws  = ">= 2.23, <= 5.0"
+    null = ">= 2.1"
+    ct = {
+      source  = "poseidon/ct"
+      version = "~> 0.9"
+    }
  }
 }
--- a/aws/fedora-coreos/kubernetes/workers.tf
+++ b/aws/fedora-coreos/kubernetes/workers.tf
@ -9,6 +9,7 @@ module "workers" {
  worker_count    = var.worker_count
  instance_type   = var.worker_type
  os_stream       = var.os_stream
+  arch            = var.arch
  disk_size       = var.disk_size
  spot_price      = var.worker_price
  target_groups   = var.worker_target_groups
--- a/aws/fedora-coreos/kubernetes/workers/ami.tf
+++ b/aws/fedora-coreos/kubernetes/workers/ami.tf
@ -1,3 +1,6 @@
+locals {
+  ami_id = var.arch == "arm64" ? data.aws_ami.fedora-coreos-arm[0].image_id : data.aws_ami.fedora-coreos.image_id
+}

 data "aws_ami" "fedora-coreos" {
  most_recent = true
@ -18,3 +21,25 @@ data "aws_ami" "fedora-coreos" {
    values = ["Fedora CoreOS ${var.os_stream} *"]
  }
 }
+
+data "aws_ami" "fedora-coreos-arm" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["125523088429"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "description"
+    values = ["Fedora CoreOS ${var.os_stream} *"]
+  }
+}
--- a/aws/fedora-coreos/kubernetes/workers/butane/worker.yaml
+++ b/aws/fedora-coreos/kubernetes/workers/butane/worker.yaml
@ -1,10 +1,12 @@
 ---
 variant: fcos
-version: 1.0.0
+version: 1.4.0
 systemd:
  units:
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -23,10 +25,13 @@ systemd:
      contents: |
        [Unit]
        Description=Kubelet (System Container)
+        Requires=afterburn.service
+        After=afterburn.service
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.6
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        EnvironmentFile=/run/metadata/afterburn
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -34,64 +39,44 @@ systemd:
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/podman rm kubelet
        ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /etc/machine-id:/etc/machine-id:ro \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
          $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
-          --cgroups-per-qos=true \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
+          --config=/etc/kubernetes/kubelet.yaml \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
          %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
+          --provider-id=aws:///$${AFTERBURN_AWS_AVAILABILITY_ZONE}/$${AFTERBURN_AWS_INSTANCE_ID}
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
-    - name: delete-node.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Delete Kubernetes node on shutdown
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.6 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
-        [Install]
-        WantedBy=multi-user.target
 storage:
  directories:
    - path: /etc/kubernetes
@ -101,6 +86,38 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: ${cluster_domain_suffix}
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -110,6 +127,13 @@ storage:
        inline: |
          net.ipv4.conf.default.rp_filter=0
          net.ipv4.conf.*.rp_filter=0
+    - path: /etc/systemd/network/50-flannel.link
+      contents:
+        inline: |
+          [Match]
+          OriginalName=flannel*
+          [Link]
+          MACAddressPolicy=none
    - path: /etc/systemd/system.conf.d/accounting.conf
      contents:
        inline: |
@ -117,9 +141,27 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ${ssh_authorized_key}
-
--- a/aws/fedora-coreos/kubernetes/workers/variables.tf
+++ b/aws/fedora-coreos/kubernetes/workers/variables.tf
@ -36,20 +36,25 @@ variable "instance_type" {

 variable "os_stream" {
  type        = string
-  description = "Fedora CoreOs image stream for instances (e.g. stable, testing, next)"
+  description = "Fedora CoreOS image stream for instances (e.g. stable, testing, next)"
  default     = "stable"
+
+  validation {
+    condition     = contains(["stable", "testing", "next"], var.os_stream)
+    error_message = "The os_stream must be stable, testing, or next."
+  }
 }

 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
@ -72,7 +77,7 @@ variable "target_groups" {

 variable "snippets" {
  type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
  default     = []
 }

@ -108,3 +113,22 @@ variable "node_labels" {
  description = "List of initial node labels"
  default     = []
 }
+
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
--- a/aws/fedora-coreos/kubernetes/workers/versions.tf
+++ b/aws/fedora-coreos/kubernetes/workers/versions.tf
@ -1,4 +1,12 @@
+# Terraform version and plugin versions

 terraform {
-  required_version = ">= 0.12"
+  required_version = ">= 0.13.0, < 2.0.0"
+  required_providers {
+    aws = ">= 2.23, <= 5.0"
+    ct = {
+      source  = "poseidon/ct"
+      version = "~> 0.9"
+    }
+  }
 }
--- a/aws/fedora-coreos/kubernetes/workers/workers.tf
+++ b/aws/fedora-coreos/kubernetes/workers/workers.tf
@ -1,6 +1,6 @@
 # Workers AutoScaling Group
 resource "aws_autoscaling_group" "workers" {
-  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
+  name = "${var.name}-worker"

  # count
  desired_capacity          = var.worker_count
@ -13,7 +13,10 @@ resource "aws_autoscaling_group" "workers" {
  vpc_zone_identifier = var.subnet_ids

  # template
-  launch_configuration = aws_launch_configuration.worker.name
+  launch_template {
+    id      = aws_launch_template.worker.id
+    version = aws_launch_template.worker.latest_version
+  }

  # target groups to which instances should be added
  target_group_arns = flatten([
@ -22,6 +25,14 @@ resource "aws_autoscaling_group" "workers" {
    var.target_groups,
  ])

+  instance_refresh {
+    strategy = "Rolling"
+    preferences {
+      instance_warmup        = 120
+      min_healthy_percentage = 90
+    }
+  }
+
  lifecycle {
    # override the default destroy and replace update behavior
    create_before_destroy = true
@ -33,34 +44,50 @@ resource "aws_autoscaling_group" "workers" {
  # used. Disable wait to avoid issues and align with other clouds.
  wait_for_capacity_timeout = "0"

-  tags = [
-    {
-      key                 = "Name"
-      value               = "${var.name}-worker"
-      propagate_at_launch = true
-    },
-  ]
+  tag {
+    key                 = "Name"
+    value               = "${var.name}-worker"
+    propagate_at_launch = true
+  }
 }

 # Worker template
-resource "aws_launch_configuration" "worker" {
-  image_id          = data.aws_ami.fedora-coreos.image_id
-  instance_type     = var.instance_type
-  spot_price        = var.spot_price > 0 ? var.spot_price : null
-  enable_monitoring = false
+resource "aws_launch_template" "worker" {
+  name_prefix   = "${var.name}-worker"
+  image_id      = local.ami_id
+  instance_type = var.instance_type
+  monitoring {
+    enabled = false
+  }

-  user_data = data.ct_config.worker-ignition.rendered
+  user_data = sensitive(base64encode(data.ct_config.worker.rendered))

  # storage
-  root_block_device {
-    volume_type = var.disk_type
-    volume_size = var.disk_size
-    iops        = var.disk_iops
-    encrypted   = true
+  ebs_optimized = true
+  block_device_mappings {
+    device_name = "/dev/xvda"
+    ebs {
+      volume_type           = var.disk_type
+      volume_size           = var.disk_size
+      iops                  = var.disk_iops
+      encrypted             = true
+      delete_on_termination = true
+    }
  }

  # network
-  security_groups = var.security_groups
+  vpc_security_group_ids = var.security_groups
+
+  # spot
+  dynamic "instance_market_options" {
+    for_each = var.spot_price > 0 ? [1] : []
+    content {
+      market_type = "spot"
+      spot_options {
+        max_price = var.spot_price
+      }
+    }
+  }

  lifecycle {
    // Override the default destroy and replace update behavior
@ -69,23 +96,16 @@ resource "aws_launch_configuration" "worker" {
  }
 }

-# Worker Ignition config
-data "ct_config" "worker-ignition" {
-  content  = data.template_file.worker-config.rendered
-  strict   = true
-  snippets = var.snippets
-}
-
-# Worker Fedora CoreOS config
-data "template_file" "worker-config" {
-  template = file("${path.module}/fcc/worker.yaml")
-
-  vars = {
+# Fedora CoreOS worker
+data "ct_config" "worker" {
+  content = templatefile("${path.module}/butane/worker.yaml", {
    kubeconfig             = indent(10, var.kubeconfig)
    ssh_authorized_key     = var.ssh_authorized_key
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
    node_labels            = join(",", var.node_labels)
-  }
+    node_taints            = join(",", var.node_taints)
+  })
+  strict   = true
+  snippets = var.snippets
 }
-
--- a/aws/container-linux/kubernetes/LICENSE
+++ b/aws/container-linux/kubernetes/LICENSE
--- a/aws/container-linux/kubernetes/README.md
+++ b/aws/container-linux/kubernetes/README.md
@ -11,13 +11,13 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.6 (upstream)
+* Kubernetes v1.26.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/cl/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#container-linux) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot](https://typhoon.psdn.io/flatcar-linux/aws/#spot) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, CSI, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs

-Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/cl/aws/).
+Please see the [official docs](https://typhoon.psdn.io) and the AWS [tutorial](https://typhoon.psdn.io/flatcar-linux/aws/).

--- a/aws/container-linux/kubernetes/ami.tf
+++ b/aws/container-linux/kubernetes/ami.tf
@ -1,33 +1,10 @@
 locals {
-  # Pick a CoreOS Container Linux derivative
-  # coreos-stable -> Container Linux AMI
+  # Pick a Flatcar Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id
-
-  flavor  = split("-", var.os_image)[0]
+  ami_id  = var.arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
  channel = split("-", var.os_image)[1]
 }

-data "aws_ami" "coreos" {
-  most_recent = true
-  owners      = ["595879546273"]
-
-  filter {
-    name   = "architecture"
-    values = ["x86_64"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "name"
-    values = ["CoreOS-${local.flavor == "coreos" ? local.channel : "stable"}-*"]
-  }
-}
-
 data "aws_ami" "flatcar" {
  most_recent = true
  owners      = ["075585003325"]
@ -44,7 +21,29 @@ data "aws_ami" "flatcar" {

  filter {
    name   = "name"
-    values = ["Flatcar-${local.flavor == "flatcar" ? local.channel : "stable"}-*"]
+    values = ["Flatcar-${local.channel}-*"]
+  }
+}
+
+data "aws_ami" "flatcar-arm64" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
  }
 }

--- a/aws/container-linux/kubernetes/bootstrap.tf
+++ b/aws/container-linux/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=835890025bfb3499aa0cd5a9704e9e7d3e7e5fe5"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4621c6b256129d1ede32ae1f72bcc1473664cb33"

  cluster_name          = var.cluster_name
  api_servers           = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers          = aws_route53_record.etcds.*.fqdn
-  asset_dir             = var.asset_dir
  networking            = var.networking
  network_mtu           = var.network_mtu
  pod_cidr              = var.pod_cidr
@ -13,5 +12,6 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
+  daemonset_tolerations = var.daemonset_tolerations
 }

--- a/aws/flatcar-linux/kubernetes/butane/controller.yaml
+++ b/aws/flatcar-linux/kubernetes/butane/controller.yaml
@ -0,0 +1,231 @@
+variant: flatcar
+version: 1.0.0
+systemd:
+  units:
+    - name: etcd-member.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=etcd (System Container)
+        Documentation=https://github.com/etcd-io/etcd
+        Requires=docker.service
+        After=docker.service
+        [Service]
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.7
+        ExecStartPre=/usr/bin/docker run -d \
+          --name etcd \
+          --network host \
+          --env-file /etc/etcd/etcd.env \
+          --user 232:232 \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro \
+          --volume /var/lib/etcd:/var/lib/etcd:rw \
+          $${ETCD_IMAGE}
+        ExecStart=docker logs -f etcd
+        ExecStop=docker stop etcd
+        ExecStopPost=docker rm etcd
+        Restart=always
+        RestartSec=10s
+        TimeoutStartSec=0
+        LimitNOFILE=40000
+        [Install]
+        WantedBy=multi-user.target
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+        RequiredBy=etcd-member.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet (System Container)
+        Requires=docker.service
+        After=docker.service
+        Requires=coreos-metadata.service
+        After=coreos-metadata.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        EnvironmentFile=/run/metadata/coreos
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/containerd:/var/lib/containerd \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --config=/etc/kubernetes/kubelet.yaml \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --node-labels=node.kubernetes.io/controller="true" \
+          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID} \
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=10
+        [Install]
+        WantedBy=multi-user.target
+    - name: bootstrap.service
+      contents: |
+        [Unit]
+        Description=Kubernetes control plane
+        Wants=docker.service
+        After=docker.service
+        ConditionPathExists=!/opt/bootstrap/bootstrap.done
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        WorkingDirectory=/opt/bootstrap
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        ExecStart=/usr/bin/docker run \
+            -v /etc/kubernetes/pki:/etc/kubernetes/pki:ro \
+            -v /opt/bootstrap/assets:/assets:ro \
+            -v /opt/bootstrap/apply:/apply:ro \
+            --entrypoint=/apply \
+            $${KUBELET_IMAGE}
+        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  directories:
+    - path: /var/lib/etcd
+      mode: 0700
+      overwrite: true
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: ${cluster_domain_suffix}
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
+    - path: /opt/bootstrap/layout
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
+          awk '/#####/ {filename=$2; next} {print > filename}' assets
+          mkdir -p /etc/ssl/etcd/etcd
+          mkdir -p /etc/kubernetes/pki
+          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
+          chown -R etcd:etcd /etc/ssl/etcd
+          chmod -R 500 /etc/ssl/etcd
+          chmod -R 700 /var/lib/etcd
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
+          mkdir -p /etc/kubernetes/manifests
+          mv static-manifests/* /etc/kubernetes/manifests/
+          mkdir -p /opt/bootstrap/assets
+          mv manifests /opt/bootstrap/assets/manifests
+          mv manifests-networking/* /opt/bootstrap/assets/manifests/
+          rm -rf assets auth static-manifests tls manifests-networking
+    - path: /opt/bootstrap/apply
+      mode: 0544
+      contents:
+        inline: |
+          #!/bin/bash -e
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
+          until kubectl version; do
+            echo "Waiting for static pod control plane"
+            sleep 5
+          done
+          until kubectl apply -f /assets/manifests -R; do
+             echo "Retry applying manifests"
+             sleep 5
+          done
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
+    - path: /etc/sysctl.d/max-user-watches.conf
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+    - path: /etc/etcd/etcd.env
+      mode: 0644
+      contents:
+          inline: |
+            ETCD_NAME=${etcd_name}
+            ETCD_DATA_DIR=/var/lib/etcd
+            ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
+            ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380
+            ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
+            ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
+            ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381
+            ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}
+            ETCD_STRICT_RECONFIG_CHECK=true
+            ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt
+            ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt
+            ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key
+            ETCD_CLIENT_CERT_AUTH=true
+            ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt
+            ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
+            ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
+            ETCD_PEER_CLIENT_CERT_AUTH=true
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/controllers.tf
+++ b/aws/container-linux/kubernetes/controllers.tf
@ -24,7 +24,7 @@ resource "aws_instance" "controllers" {
  instance_type = var.controller_type

  ami       = local.ami_id
-  user_data = data.ct_config.controller-ignitions.*.rendered[count.index]
+  user_data = data.ct_config.controllers.*.rendered[count.index]

  # storage
  root_block_device {
@ -32,6 +32,7 @@ resource "aws_instance" "controllers" {
    volume_size = var.disk_size
    iops        = var.disk_iops
    encrypted   = true
+    tags        = {}
  }

  # network
@ -47,42 +48,22 @@ resource "aws_instance" "controllers" {
  }
 }

-# Controller Ignition configs
-data "ct_config" "controller-ignitions" {
-  count    = var.controller_count
-  content  = data.template_file.controller-configs.*.rendered[count.index]
-  strict   = true
-  snippets = var.controller_snippets
-}
-
-# Controller Container Linux configs
-data "template_file" "controller-configs" {
+# Flatcar Linux controllers
+data "ct_config" "controllers" {
  count = var.controller_count
-
-  template = file("${path.module}/cl/controller.yaml")
-
-  vars = {
+  content = templatefile("${path.module}/butane/controller.yaml", {
    # Cannot use cyclic dependencies on controllers or their DNS records
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
-    cgroup_driver          = local.flavor == "flatcar" && local.channel == "edge" ? "systemd" : "cgroupfs"
+    etcd_initial_cluster = join(",", [
+      for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
+    ])
    kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
    ssh_authorized_key     = var.ssh_authorized_key
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
-  }
+  })
+  strict   = true
+  snippets = var.controller_snippets
 }
-
-data "template_file" "etcds" {
-  count    = var.controller_count
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = count.index
-    cluster_name = var.cluster_name
-    dns_zone     = var.dns_zone
-  }
-}
-
--- a/aws/container-linux/kubernetes/network.tf
+++ b/aws/container-linux/kubernetes/network.tf
--- a/aws/container-linux/kubernetes/nlb.tf
+++ b/aws/container-linux/kubernetes/nlb.tf
@ -17,6 +17,7 @@ resource "aws_route53_record" "apiserver" {
 resource "aws_lb" "nlb" {
  name               = "${var.cluster_name}-nlb"
  load_balancer_type = "network"
+  ip_address_type    = "dualstack"
  internal           = false

  subnets = aws_subnet.public.*.id
--- a/aws/container-linux/kubernetes/outputs.tf
+++ b/aws/container-linux/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "worker_security_groups" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom load balancing
@ -52,3 +54,10 @@ output "worker_target_group_https" {
  value       = module.workers.target_group_https
 }

+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/aws/container-linux/kubernetes/security.tf
+++ b/aws/container-linux/kubernetes/security.tf
@ -201,8 +201,8 @@ resource "aws_security_group_rule" "controller-scheduler-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10251
-  to_port                  = 10251
+  from_port                = 10259
+  to_port                  = 10259
  source_security_group_id = aws_security_group.worker.id
 }

@ -212,8 +212,8 @@ resource "aws_security_group_rule" "controller-manager-metrics" {

  type                     = "ingress"
  protocol                 = "tcp"
-  from_port                = 10252
-  to_port                  = 10252
+  from_port                = 10257
+  to_port                  = 10257
  source_security_group_id = aws_security_group.worker.id
 }

--- a/aws/container-linux/kubernetes/ssh.tf
+++ b/aws/container-linux/kubernetes/ssh.tf
@ -24,7 +24,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/aws/container-linux/kubernetes/variables.tf
+++ b/aws/container-linux/kubernetes/variables.tf
@ -43,26 +43,31 @@ variable "worker_type" {

 variable "os_image" {
  type        = string
-  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
  default     = "flatcar-stable"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, or flatcar-alpha."
+  }
 }

 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
  type        = number
-  description = "IOPS of the EBS volume (e.g. 100)"
-  default     = 0
+  description = "IOPS of the EBS volume (e.g. 3000)"
+  default     = 3000
 }

 variable "worker_price" {
@ -98,8 +103,8 @@ variable "ssh_authorized_key" {

 variable "networking" {
  type        = string
-  description = "Choice of networking provider (calico or flannel)"
-  default     = "calico"
+  description = "Choice of networking provider (flannel, calico, or cilium)"
+  default     = "cilium"
 }

 variable "network_mtu" {
@ -137,8 +142,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -149,15 +154,25 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by CoreDNS. Default is cluster.local (e.g. foo.default.svc.cluster.local)"
  default     = "cluster.local"
 }

+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
+
+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
--- a/aws/flatcar-linux/kubernetes/versions.tf
+++ b/aws/flatcar-linux/kubernetes/versions.tf
@ -0,0 +1,13 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = ">= 0.13.0, < 2.0.0"
+  required_providers {
+    aws  = ">= 2.23, <= 5.0"
+    null = ">= 2.1"
+    ct = {
+      source  = "poseidon/ct"
+      version = "~> 0.11"
+    }
+  }
+}
--- a/aws/container-linux/kubernetes/workers.tf
+++ b/aws/container-linux/kubernetes/workers.tf
@ -9,6 +9,7 @@ module "workers" {
  worker_count    = var.worker_count
  instance_type   = var.worker_type
  os_image        = var.os_image
+  arch            = var.arch
  disk_size       = var.disk_size
  spot_price      = var.worker_price
  target_groups   = var.worker_target_groups
--- a/aws/container-linux/kubernetes/workers/ami.tf
+++ b/aws/container-linux/kubernetes/workers/ami.tf
@ -1,33 +1,10 @@
 locals {
-  # Pick a CoreOS Container Linux derivative
-  # coreos-stable -> Container Linux AMI
+  # Pick a Flatcar Linux AMI
  # flatcar-stable -> Flatcar Linux AMI
-  ami_id = local.flavor == "flatcar" ? data.aws_ami.flatcar.image_id : data.aws_ami.coreos.image_id
-
-  flavor  = split("-", var.os_image)[0]
+  ami_id  = var.arch == "arm64" ? data.aws_ami.flatcar-arm64[0].image_id : data.aws_ami.flatcar.image_id
  channel = split("-", var.os_image)[1]
 }

-data "aws_ami" "coreos" {
-  most_recent = true
-  owners      = ["595879546273"]
-
-  filter {
-    name   = "architecture"
-    values = ["x86_64"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "name"
-    values = ["CoreOS-${local.flavor == "coreos" ? local.channel : "stable"}-*"]
-  }
-}
-
 data "aws_ami" "flatcar" {
  most_recent = true
  owners      = ["075585003325"]
@ -44,7 +21,28 @@ data "aws_ami" "flatcar" {

  filter {
    name   = "name"
-    values = ["Flatcar-${local.flavor == "flatcar" ? local.channel : "stable"}-*"]
+    values = ["Flatcar-${local.channel}-*"]
  }
 }

+data "aws_ami" "flatcar-arm64" {
+  count = var.arch == "arm64" ? 1 : 0
+
+  most_recent = true
+  owners      = ["075585003325"]
+
+  filter {
+    name   = "architecture"
+    values = ["arm64"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["Flatcar-${local.channel}-*"]
+  }
+}
--- a/aws/flatcar-linux/kubernetes/workers/butane/worker.yaml
+++ b/aws/flatcar-linux/kubernetes/workers/butane/worker.yaml
@ -0,0 +1,129 @@
+variant: flatcar
+version: 1.0.0
+systemd:
+  units:
+    - name: docker.service
+      enabled: true
+    - name: locksmithd.service
+      mask: true
+    - name: wait-for-dns.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Wait for DNS entries
+        Wants=systemd-resolved.service
+        Before=kubelet.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=true
+        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
+        [Install]
+        RequiredBy=kubelet.service
+    - name: kubelet.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubelet
+        Requires=docker.service
+        After=docker.service
+        Requires=coreos-metadata.service
+        After=coreos-metadata.service
+        Wants=rpc-statd.service
+        [Service]
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        EnvironmentFile=/run/metadata/coreos
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
+        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
+        ExecStartPre=/bin/mkdir -p /opt/cni/bin
+        ExecStartPre=/bin/mkdir -p /var/lib/calico
+        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
+        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
+        # Podman, rkt, or runc run container processes, whereas docker run
+        # is a client to a daemon and requires workarounds to use within a
+        # systemd unit. https://github.com/moby/moby/issues/6791
+        ExecStartPre=/usr/bin/docker run -d \
+          --name kubelet \
+          --privileged \
+          --pid host \
+          --network host \
+          -v /etc/cni/net.d:/etc/cni/net.d:ro \
+          -v /etc/kubernetes:/etc/kubernetes:ro \
+          -v /etc/machine-id:/etc/machine-id:ro \
+          -v /usr/lib/os-release:/etc/os-release:ro \
+          -v /lib/modules:/lib/modules:ro \
+          -v /run:/run \
+          -v /sys/fs/cgroup:/sys/fs/cgroup \
+          -v /var/lib/calico:/var/lib/calico:ro \
+          -v /var/lib/containerd:/var/lib/containerd \
+          -v /var/lib/kubelet:/var/lib/kubelet:rshared \
+          -v /var/log:/var/log \
+          -v /opt/cni/bin:/opt/cni/bin \
+          $${KUBELET_IMAGE} \
+          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
+          --config=/etc/kubernetes/kubelet.yaml \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
+          --kubeconfig=/var/lib/kubelet/kubeconfig \
+          --node-labels=node.kubernetes.io/node \
+          %{~ for label in split(",", node_labels) ~}
+          --node-labels=${label} \
+          %{~ endfor ~}
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
+          --provider-id=aws:///$${COREOS_EC2_AVAILABILITY_ZONE}/$${COREOS_EC2_INSTANCE_ID}
+        ExecStart=docker logs -f kubelet
+        ExecStop=docker stop kubelet
+        ExecStopPost=docker rm kubelet
+        Restart=always
+        RestartSec=5
+        [Install]
+        WantedBy=multi-user.target
+storage:
+  files:
+    - path: /etc/kubernetes/kubeconfig
+      mode: 0644
+      contents:
+        inline: |
+          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: ${cluster_domain_suffix}
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
+    - path: /etc/sysctl.d/max-user-watches.conf
+      mode: 0644
+      contents:
+        inline: |
+          fs.inotify.max_user_watches=16184
+passwd:
+  users:
+    - name: core
+      ssh_authorized_keys:
+        - "${ssh_authorized_key}"
--- a/aws/container-linux/kubernetes/workers/ingress.tf
+++ b/aws/container-linux/kubernetes/workers/ingress.tf
--- a/aws/container-linux/kubernetes/workers/outputs.tf
+++ b/aws/container-linux/kubernetes/workers/outputs.tf
--- a/aws/container-linux/kubernetes/workers/variables.tf
+++ b/aws/container-linux/kubernetes/workers/variables.tf
@ -36,20 +36,25 @@ variable "instance_type" {

 variable "os_image" {
  type        = string
-  description = "AMI channel for a Container Linux derivative (coreos-stable, coreos-beta, coreos-alpha, flatcar-stable, flatcar-beta, flatcar-alpha, flatcar-edge)"
+  description = "AMI channel for a Container Linux derivative (flatcar-stable, flatcar-beta, flatcar-alpha)"
  default     = "flatcar-stable"
+
+  validation {
+    condition     = contains(["flatcar-stable", "flatcar-beta", "flatcar-alpha"], var.os_image)
+    error_message = "The os_image must be flatcar-stable, flatcar-beta, or flatcar-alpha."
+  }
 }

 variable "disk_size" {
  type        = number
  description = "Size of the EBS volume in GB"
-  default     = 40
+  default     = 30
 }

 variable "disk_type" {
  type        = string
-  description = "Type of the EBS volume (e.g. standard, gp2, io1)"
-  default     = "gp2"
+  description = "Type of the EBS volume (e.g. standard, gp2, gp3, io1)"
+  default     = "gp3"
 }

 variable "disk_iops" {
@ -108,3 +113,22 @@ variable "node_labels" {
  description = "List of initial node labels"
  default     = []
 }
+
+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
+# unofficial, undocumented, unsupported
+
+variable "arch" {
+  type        = string
+  description = "Container architecture (amd64 or arm64)"
+  default     = "amd64"
+
+  validation {
+    condition     = var.arch == "amd64" || var.arch == "arm64"
+    error_message = "The arch must be amd64 or arm64."
+  }
+}
--- a/aws/flatcar-linux/kubernetes/workers/versions.tf
+++ b/aws/flatcar-linux/kubernetes/workers/versions.tf
@ -0,0 +1,12 @@
+# Terraform version and plugin versions
+
+terraform {
+  required_version = ">= 0.13.0, < 2.0.0"
+  required_providers {
+    aws = ">= 2.23, <= 5.0"
+    ct = {
+      source  = "poseidon/ct"
+      version = "~> 0.11"
+    }
+  }
+}
--- a/aws/container-linux/kubernetes/workers/workers.tf
+++ b/aws/container-linux/kubernetes/workers/workers.tf
@ -1,6 +1,6 @@
 # Workers AutoScaling Group
 resource "aws_autoscaling_group" "workers" {
-  name = "${var.name}-worker ${aws_launch_configuration.worker.name}"
+  name = "${var.name}-worker"

  # count
  desired_capacity          = var.worker_count
@ -13,7 +13,10 @@ resource "aws_autoscaling_group" "workers" {
  vpc_zone_identifier = var.subnet_ids

  # template
-  launch_configuration = aws_launch_configuration.worker.name
+  launch_template {
+    id      = aws_launch_template.worker.id
+    version = aws_launch_template.worker.latest_version
+  }

  # target groups to which instances should be added
  target_group_arns = flatten([
@ -22,6 +25,14 @@ resource "aws_autoscaling_group" "workers" {
    var.target_groups,
  ])

+  instance_refresh {
+    strategy = "Rolling"
+    preferences {
+      instance_warmup        = 120
+      min_healthy_percentage = 90
+    }
+  }
+
  lifecycle {
    # override the default destroy and replace update behavior
    create_before_destroy = true
@ -33,34 +44,50 @@ resource "aws_autoscaling_group" "workers" {
  # used. Disable wait to avoid issues and align with other clouds.
  wait_for_capacity_timeout = "0"

-  tags = [
-    {
-      key                 = "Name"
-      value               = "${var.name}-worker"
-      propagate_at_launch = true
-    },
-  ]
+  tag {
+    key                 = "Name"
+    value               = "${var.name}-worker"
+    propagate_at_launch = true
+  }
 }

 # Worker template
-resource "aws_launch_configuration" "worker" {
-  image_id          = local.ami_id
-  instance_type     = var.instance_type
-  spot_price        = var.spot_price > 0 ? var.spot_price : null
-  enable_monitoring = false
+resource "aws_launch_template" "worker" {
+  name_prefix   = "${var.name}-worker"
+  image_id      = local.ami_id
+  instance_type = var.instance_type
+  monitoring {
+    enabled = false
+  }

-  user_data = data.ct_config.worker-ignition.rendered
+  user_data = sensitive(base64encode(data.ct_config.worker.rendered))

  # storage
-  root_block_device {
-    volume_type = var.disk_type
-    volume_size = var.disk_size
-    iops        = var.disk_iops
-    encrypted   = true
+  ebs_optimized = true
+  block_device_mappings {
+    device_name = "/dev/xvda"
+    ebs {
+      volume_type           = var.disk_type
+      volume_size           = var.disk_size
+      iops                  = var.disk_iops
+      encrypted             = true
+      delete_on_termination = true
+    }
  }

  # network
-  security_groups = var.security_groups
+  vpc_security_group_ids = var.security_groups
+
+  # spot
+  dynamic "instance_market_options" {
+    for_each = var.spot_price > 0 ? [1] : []
+    content {
+      market_type = "spot"
+      spot_options {
+        max_price = var.spot_price
+      }
+    }
+  }

  lifecycle {
    // Override the default destroy and replace update behavior
@ -69,24 +96,16 @@ resource "aws_launch_configuration" "worker" {
  }
 }

-# Worker Ignition config
-data "ct_config" "worker-ignition" {
-  content  = data.template_file.worker-config.rendered
-  strict   = true
-  snippets = var.snippets
-}
-
-# Worker Container Linux config
-data "template_file" "worker-config" {
-  template = file("${path.module}/cl/worker.yaml")
-
-  vars = {
+# Flatcar Linux worker
+data "ct_config" "worker" {
+  content = templatefile("${path.module}/butane/worker.yaml", {
    kubeconfig             = indent(10, var.kubeconfig)
    ssh_authorized_key     = var.ssh_authorized_key
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
-    cgroup_driver          = local.flavor == "flatcar" && local.channel == "edge" ? "systemd" : "cgroupfs"
    node_labels            = join(",", var.node_labels)
-  }
+    node_taints            = join(",", var.node_taints)
+  })
+  strict   = true
+  snippets = var.snippets
 }
-
--- a/azure/container-linux/kubernetes/cl/controller.yaml
+++ b/azure/container-linux/kubernetes/cl/controller.yaml
@ -1,199 +0,0 @@
---
-systemd:
-  units:
-    - name: etcd-member.service
-      enabled: true
-      dropins:
-        - name: 40-etcd-cluster.conf
-          contents: |
-            [Service]
-            Environment="ETCD_IMAGE_TAG=v3.4.9"
-            Environment="ETCD_IMAGE_URL=docker://quay.io/coreos/etcd"
-            Environment="RKT_RUN_ARGS=--insecure-options=image"
-            Environment="ETCD_NAME=${etcd_name}"
-            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379"
-            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://${etcd_domain}:2380"
-            Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
-            Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
-            Environment="ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:2381"
-            Environment="ETCD_INITIAL_CLUSTER=${etcd_initial_cluster}"
-            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
-            Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
-            Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/server-ca.crt"
-            Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd/server.crt"
-            Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd/server.key"
-            Environment="ETCD_CLIENT_CERT_AUTH=true"
-            Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/peer-ca.crt"
-            Environment="ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt"
-            Environment="ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key"
-            Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-        RequiredBy=etcd-member.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.6
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=10
-        [Install]
-        WantedBy=multi-user.target
-    - name: bootstrap.service
-      contents: |
-        [Unit]
-        Description=Kubernetes control plane
-        ConditionPathExists=!/opt/bootstrap/bootstrap.done
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        WorkingDirectory=/opt/bootstrap
-        ExecStart=/usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes/bootstrap-secrets \
-            --mount volume=config,target=/etc/kubernetes/secrets \
-            --volume assets,kind=host,source=/opt/bootstrap/assets \
-            --mount volume=assets,target=/assets \
-            --volume script,kind=host,source=/opt/bootstrap/apply \
-            --mount volume=script,target=/apply \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.6 \
-            --net=host \
-            --dns=host \
-            --exec=/apply
-        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /opt/bootstrap/layout
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
-          awk '/#####/ {filename=$2; next} {print > filename}' assets
-          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
-          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
-          chown -R etcd:etcd /etc/ssl/etcd
-          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
-          mkdir -p /etc/kubernetes/manifests
-          mv static-manifests/* /etc/kubernetes/manifests/
-          mkdir -p /opt/bootstrap/assets
-          mv manifests /opt/bootstrap/assets/manifests
-          mv manifests-networking/* /opt/bootstrap/assets/manifests/
-          rm -rf assets auth static-manifests tls manifests-networking
-    - path: /opt/bootstrap/apply
-      filesystem: root
-      mode: 0544
-      contents:
-        inline: |
-          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
-          until kubectl version; do
-            echo "Waiting for static pod control plane"
-            sleep 5
-          done
-          until kubectl apply -f /assets/manifests -R; do
-             echo "Retry applying manifests"
-             sleep 5
-          done
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/security.tf
+++ b/azure/container-linux/kubernetes/security.tf
@ -1,428 +0,0 @@
-# Controller security group
-
-resource "azurerm_network_security_group" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = "${var.cluster_name}-controller"
-  location = azurerm_resource_group.cluster.location
-}
-
-resource "azurerm_network_security_rule" "controller-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-icmp"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "1995"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Icmp"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-resource "azurerm_network_security_rule" "controller-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-ssh"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2000"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "22"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-resource "azurerm_network_security_rule" "controller-etcd" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-etcd"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2005"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "2379-2380"
-  source_address_prefix       = azurerm_subnet.controller.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-# Allow Prometheus to scrape etcd metrics
-resource "azurerm_network_security_rule" "controller-etcd-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-etcd-metrics"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2010"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "2381"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-# Allow Prometheus to scrape kube-proxy metrics
-resource "azurerm_network_security_rule" "controller-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-kube-proxy-metrics"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2011"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10249"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-# Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
-resource "azurerm_network_security_rule" "controller-kube-metrics" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-kube-metrics"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2012"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10251-10252"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-resource "azurerm_network_security_rule" "controller-apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-apiserver"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2015"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "6443"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-resource "azurerm_network_security_rule" "controller-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
-
-  name                        = "allow-cilium-health"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2019"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "4240"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-resource "azurerm_network_security_rule" "controller-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-vxlan"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2020"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "4789"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-resource "azurerm_network_security_rule" "controller-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-linux-vxlan"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2021"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "8472"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-# Allow Prometheus to scrape node-exporter daemonset
-resource "azurerm_network_security_rule" "controller-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-node-exporter"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2025"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "9100"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
-}
-
-# Allow apiserver to access kubelet's for exec, log, port-forward
-resource "azurerm_network_security_rule" "controller-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-kubelet"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2030"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10250"
-
-  # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix = azurerm_subnet.controller.address_prefix
-}
-
-# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
-# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
-
-resource "azurerm_network_security_rule" "controller-allow-loadblancer" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-loadbalancer"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "3000"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "*"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefix       = "AzureLoadBalancer"
-  destination_address_prefix  = "*"
-}
-
-resource "azurerm_network_security_rule" "controller-deny-all" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "deny-all"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "3005"
-  access                      = "Deny"
-  direction                   = "Inbound"
-  protocol                    = "*"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefix       = "*"
-  destination_address_prefix  = "*"
-}
-
-# Worker security group
-
-resource "azurerm_network_security_group" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name     = "${var.cluster_name}-worker"
-  location = azurerm_resource_group.cluster.location
-}
-
-resource "azurerm_network_security_rule" "worker-icmp" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-icmp"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "1995"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Icmp"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-resource "azurerm_network_security_rule" "worker-ssh" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-ssh"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2000"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "22"
-  source_address_prefix       = azurerm_subnet.controller.address_prefix
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-resource "azurerm_network_security_rule" "worker-http" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-http"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2005"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "80"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-resource "azurerm_network_security_rule" "worker-https" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-https"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2010"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "443"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-resource "azurerm_network_security_rule" "worker-cilium-health" {
-  resource_group_name = azurerm_resource_group.cluster.name
-  count               = var.networking == "cilium" ? 1 : 0
-
-  name                        = "allow-cilium-health"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2014"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "4240"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-resource "azurerm_network_security_rule" "worker-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-vxlan"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2015"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "4789"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-resource "azurerm_network_security_rule" "worker-linux-vxlan" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-linux-vxlan"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2016"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "8472"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-# Allow Prometheus to scrape node-exporter daemonset
-resource "azurerm_network_security_rule" "worker-node-exporter" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-node-exporter"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2020"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "9100"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-# Allow Prometheus to scrape kube-proxy
-resource "azurerm_network_security_rule" "worker-kube-proxy" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-kube-proxy"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2024"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10249"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
-}
-
-# Allow apiserver to access kubelet's for exec, log, port-forward
-resource "azurerm_network_security_rule" "worker-kubelet" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-kubelet"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2025"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10250"
-
-  # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix = azurerm_subnet.worker.address_prefix
-}
-
-# Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
-# https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#default-security-rules
-
-resource "azurerm_network_security_rule" "worker-allow-loadblancer" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "allow-loadbalancer"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "3000"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "*"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefix       = "AzureLoadBalancer"
-  destination_address_prefix  = "*"
-}
-
-resource "azurerm_network_security_rule" "worker-deny-all" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
-  name                        = "deny-all"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "3005"
-  access                      = "Deny"
-  direction                   = "Inbound"
-  protocol                    = "*"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefix       = "*"
-  destination_address_prefix  = "*"
-}
-
--- a/azure/container-linux/kubernetes/versions.tf
+++ b/azure/container-linux/kubernetes/versions.tf
@ -1,12 +0,0 @@
-# Terraform version and plugin versions
-
-terraform {
-  required_version = "~> 0.12.6"
-  required_providers {
-    azurerm  = "~> 2.8"
-    ct       = "~> 0.4"
-    template = "~> 2.1"
-    null     = "~> 2.1"
-  }
-}
-
--- a/azure/container-linux/kubernetes/workers/cl/worker.yaml
+++ b/azure/container-linux/kubernetes/workers/cl/worker.yaml
@ -1,140 +0,0 @@
---
-systemd:
-  units:
-    - name: docker.service
-      enabled: true
-    - name: locksmithd.service
-      mask: true
-    - name: wait-for-dns.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Wait for DNS entries
-        Wants=systemd-resolved.service
-        Before=kubelet.service
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
-        [Install]
-        RequiredBy=kubelet.service
-    - name: kubelet.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Kubelet
-        Wants=rpc-statd.service
-        [Service]
-        Environment=KUBELET_IMAGE=docker://quay.io/poseidon/kubelet:v1.18.6
-        Environment=KUBELET_CGROUP_DRIVER=${cgroup_driver}
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
-        ExecStartPre=/bin/mkdir -p /opt/cni/bin
-        ExecStartPre=/bin/mkdir -p /var/lib/calico
-        ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
-        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
-        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
-        ExecStart=/usr/bin/rkt run \
-          --uuid-file-save=/var/cache/kubelet-pod.uuid \
-          --stage1-from-dir=stage1-fly.aci \
-          --hosts-entry host \
-          --insecure-options=image \
-          --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=true \
-          --mount volume=etc-kubernetes,target=/etc/kubernetes \
-          --volume etc-machine-id,kind=host,source=/etc/machine-id,readOnly=true \
-          --mount volume=etc-machine-id,target=/etc/machine-id \
-          --volume etc-os-release,kind=host,source=/usr/lib/os-release,readOnly=true \
-          --mount volume=etc-os-release,target=/etc/os-release \
-          --volume=etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
-          --mount volume=etc-resolv,target=/etc/resolv.conf \
-          --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true \
-          --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
-          --volume lib-modules,kind=host,source=/lib/modules,readOnly=true \
-          --mount volume=lib-modules,target=/lib/modules \
-          --volume run,kind=host,source=/run \
-          --mount volume=run,target=/run \
-          --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
-          --mount volume=usr-share-certs,target=/usr/share/ca-certificates \
-          --volume var-lib-calico,kind=host,source=/var/lib/calico,readOnly=true \
-          --mount volume=var-lib-calico,target=/var/lib/calico \
-          --volume var-lib-docker,kind=host,source=/var/lib/docker \
-          --mount volume=var-lib-docker,target=/var/lib/docker \
-          --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,recursive=true \
-          --mount volume=var-lib-kubelet,target=/var/lib/kubelet \
-          --volume var-log,kind=host,source=/var/log \
-          --mount volume=var-log,target=/var/log \
-          --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
-          --mount volume=opt-cni-bin,target=/opt/cni/bin \
-          $${KUBELET_IMAGE} -- \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
-          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=$${KUBELET_CGROUP_DRIVER} \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
-          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/node \
-          %{~ for label in split(",", node_labels) ~}
-          --node-labels=${label} \
-          %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
-        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
-        Restart=always
-        RestartSec=5
-        [Install]
-        WantedBy=multi-user.target
-    - name: delete-node.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Waiting to delete Kubernetes node on shutdown
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/etc/kubernetes/delete-node
-        [Install]
-        WantedBy=multi-user.target
-storage:
-  files:
-    - path: /etc/kubernetes/kubeconfig
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          ${kubeconfig}
-    - path: /etc/sysctl.d/max-user-watches.conf
-      filesystem: root
-      mode: 0644
-      contents:
-        inline: |
-          fs.inotify.max_user_watches=16184
-    - path: /etc/kubernetes/delete-node
-      filesystem: root
-      mode: 0744
-      contents:
-        inline: |
-          #!/bin/bash
-          set -e
-          exec /usr/bin/rkt run \
-            --trust-keys-from-https \
-            --volume config,kind=host,source=/etc/kubernetes \
-            --mount volume=config,target=/etc/kubernetes \
-            --insecure-options=image \
-            docker://quay.io/poseidon/kubelet:v1.18.6 \
-            --net=host \
-            --dns=host \
-            --exec=/usr/local/bin/kubectl -- --kubeconfig=/etc/kubernetes/kubeconfig delete node $(hostname | tr '[:upper:]' '[:lower:]')
-passwd:
-  users:
-    - name: core
-      ssh_authorized_keys:
-        - "${ssh_authorized_key}"
--- a/azure/container-linux/kubernetes/workers/versions.tf
+++ b/azure/container-linux/kubernetes/workers/versions.tf
@ -1,4 +0,0 @@
-
-terraform {
-  required_version = ">= 0.12"
-}
--- a/azure/fedora-coreos/kubernetes/README.md
+++ b/azure/fedora-coreos/kubernetes/README.md
@ -11,10 +11,10 @@ Typhoon distributes upstream Kubernetes, architectural conventions, and cluster

 ## Features <a href="https://www.cncf.io/certification/software-conformance/"><img align="right" src="https://storage.googleapis.com/poseidon/certified-kubernetes.png"></a>

-* Kubernetes v1.18.6 (upstream)
+* Kubernetes v1.26.1 (upstream)
 * Single or multi-master, [Calico](https://www.projectcalico.org/) or [Cilium](https://github.com/cilium/cilium) or [flannel](https://github.com/coreos/flannel) networking
 * On-cluster etcd with TLS, [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/)-enabled, [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), SELinux enforcing
-* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/) customization
+* Advanced features like [worker pools](https://typhoon.psdn.io/advanced/worker-pools/), [spot priority](https://typhoon.psdn.io/fedora-coreos/azure/#low-priority) workers, and [snippets](https://typhoon.psdn.io/advanced/customization/#hosts) customization
 * Ready for Ingress, Prometheus, Grafana, and other optional [addons](https://typhoon.psdn.io/addons/overview/)

 ## Docs
--- a/azure/fedora-coreos/kubernetes/bootstrap.tf
+++ b/azure/fedora-coreos/kubernetes/bootstrap.tf
@ -1,11 +1,10 @@
 # Kubernetes assets (kubeconfig, manifests)
 module "bootstrap" {
-  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=835890025bfb3499aa0cd5a9704e9e7d3e7e5fe5"
+  source = "git::https://github.com/poseidon/terraform-render-bootstrap.git?ref=4621c6b256129d1ede32ae1f72bcc1473664cb33"

  cluster_name = var.cluster_name
  api_servers  = [format("%s.%s", var.cluster_name, var.dns_zone)]
  etcd_servers = formatlist("%s.%s", azurerm_dns_a_record.etcds.*.name, var.dns_zone)
-  asset_dir    = var.asset_dir

  networking = var.networking

@ -19,8 +18,6 @@ module "bootstrap" {
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation
-
-  # Fedora CoreOS
-  trusted_certs_dir = "/etc/pki/tls/certs"
+  daemonset_tolerations = var.daemonset_tolerations
 }

--- a/azure/fedora-coreos/kubernetes/butane/controller.yaml
+++ b/azure/fedora-coreos/kubernetes/butane/controller.yaml
@ -1,6 +1,6 @@
 ---
 variant: fcos
-version: 1.0.0
+version: 1.4.0
 systemd:
  units:
    - name: etcd-member.service
@ -8,32 +8,32 @@ systemd:
      contents: |
        [Unit]
        Description=etcd (System Container)
-        Documentation=https://github.com/coreos/etcd
-        Wants=network-online.target network.target
+        Documentation=https://github.com/etcd-io/etcd
+        Wants=network-online.target
        After=network-online.target
        [Service]
-        # https://github.com/opencontainers/runc/pull/1807
-        # Type=notify
-        # NotifyAccess=exec
+        Environment=ETCD_IMAGE=quay.io/coreos/etcd:v3.5.7
        Type=exec
+        ExecStartPre=/bin/mkdir -p /var/lib/etcd
+        ExecStartPre=-/usr/bin/podman rm etcd
+        ExecStart=/usr/bin/podman run --name etcd \
+          --env-file /etc/etcd/etcd.env \
+          --log-driver k8s-file \
+          --network host \
+          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
+          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
+          $${ETCD_IMAGE}
+        ExecStop=/usr/bin/podman stop etcd
        Restart=on-failure
        RestartSec=10s
        TimeoutStartSec=0
        LimitNOFILE=40000
-        ExecStartPre=/bin/mkdir -p /var/lib/etcd
-        ExecStartPre=-/usr/bin/podman rm etcd
-        #--volume $${NOTIFY_SOCKET}:/run/systemd/notify \
-        ExecStart=/usr/bin/podman run --name etcd \
-          --env-file /etc/etcd/etcd.env \
-          --network host \
-          --volume /var/lib/etcd:/var/lib/etcd:rw,Z \
-          --volume /etc/ssl/etcd:/etc/ssl/certs:ro,Z \
-          quay.io/coreos/etcd:v3.4.9
-        ExecStop=/usr/bin/podman stop etcd
        [Install]
        WantedBy=multi-user.target
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -54,8 +54,8 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.6
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -63,44 +63,32 @@ systemd:
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/podman rm kubelet
        ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
+          --volume /etc/machine-id:/etc/machine-id:ro \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
          $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
-          --cgroups-per-qos=true \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
+          --config=/etc/kubernetes/kubelet.yaml \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
          --node-labels=node.kubernetes.io/controller="true" \
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+          --register-with-taints=node-role.kubernetes.io/controller=:NoSchedule
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
        Restart=always
@ -119,15 +107,17 @@ systemd:
        ExecStartPre=-/usr/bin/podman rm bootstrap
        ExecStart=/usr/bin/podman run --name bootstrap \
            --network host \
-            --volume /etc/kubernetes/bootstrap-secrets:/etc/kubernetes/secrets:ro,z \
+            --volume /etc/kubernetes/pki:/etc/kubernetes/pki:ro,z \
            --volume /opt/bootstrap/assets:/assets:ro,Z \
            --volume /opt/bootstrap/apply:/apply:ro,Z \
            --entrypoint=/apply \
-            quay.io/poseidon/kubelet:v1.18.6
+            quay.io/poseidon/kubelet:v1.26.1
        ExecStartPost=/bin/touch /opt/bootstrap/bootstrap.done
        ExecStartPost=-/usr/bin/podman stop bootstrap
 storage:
  directories:
+    - path: /var/lib/etcd
+      mode: 0700
    - path: /etc/kubernetes
    - path: /opt/bootstrap
  files:
@ -136,6 +126,33 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: ${cluster_domain_suffix}
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
    - path: /opt/bootstrap/layout
      mode: 0544
      contents:
@ -144,25 +161,26 @@ storage:
          mkdir -p -- auth tls/etcd tls/k8s static-manifests manifests/coredns manifests-networking
          awk '/#####/ {filename=$2; next} {print > filename}' assets
          mkdir -p /etc/ssl/etcd/etcd
-          mkdir -p /etc/kubernetes/bootstrap-secrets
+          mkdir -p /etc/kubernetes/pki
          mv tls/etcd/{peer*,server*} /etc/ssl/etcd/etcd/
-          mv tls/etcd/etcd-client* /etc/kubernetes/bootstrap-secrets/
+          mv tls/etcd/etcd-client* /etc/kubernetes/pki/
          chown -R etcd:etcd /etc/ssl/etcd
          chmod -R 500 /etc/ssl/etcd
-          mv auth/kubeconfig /etc/kubernetes/bootstrap-secrets/
-          mv tls/k8s/* /etc/kubernetes/bootstrap-secrets/
+          mv auth/* /etc/kubernetes/pki/
+          mv tls/k8s/* /etc/kubernetes/pki/
          mkdir -p /etc/kubernetes/manifests
          mv static-manifests/* /etc/kubernetes/manifests/
          mkdir -p /opt/bootstrap/assets
          mv manifests /opt/bootstrap/assets/manifests
          mv manifests-networking/* /opt/bootstrap/assets/manifests/
          rm -rf assets auth static-manifests tls manifests-networking
+          chcon -R -u system_u -t container_file_t /etc/kubernetes/pki
    - path: /opt/bootstrap/apply
      mode: 0544
      contents:
        inline: |
          #!/bin/bash -e
-          export KUBECONFIG=/etc/kubernetes/secrets/kubeconfig
+          export KUBECONFIG=/etc/kubernetes/pki/admin.conf
          until kubectl version; do
            echo "Waiting for static pod control plane"
            sleep 5
@ -171,6 +189,11 @@ storage:
             echo "Retry applying manifests"
             sleep 5
          done
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -180,6 +203,13 @@ storage:
        inline: |
          net.ipv4.conf.default.rp_filter=0
          net.ipv4.conf.*.rp_filter=0
+    - path: /etc/systemd/network/50-flannel.link
+      contents:
+        inline: |
+          [Match]
+          OriginalName=flannel*
+          [Link]
+          MACAddressPolicy=none
    - path: /etc/systemd/system.conf.d/accounting.conf
      contents:
        inline: |
@ -191,8 +221,6 @@ storage:
      mode: 0644
      contents:
        inline: |
-          # TODO: Use a systemd dropin once podman v1.4.5 is avail.
-          NOTIFY_SOCKET=/run/systemd/notify
          ETCD_NAME=${etcd_name}
          ETCD_DATA_DIR=/var/lib/etcd
          ETCD_ADVERTISE_CLIENT_URLS=https://${etcd_domain}:2379
@ -210,6 +238,25 @@ storage:
          ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd/peer.crt
          ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd/peer.key
          ETCD_PEER_CLIENT_CERT_AUTH=true
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
--- a/azure/fedora-coreos/kubernetes/controllers.tf
+++ b/azure/fedora-coreos/kubernetes/controllers.tf
@ -35,7 +35,7 @@ resource "azurerm_linux_virtual_machine" "controllers" {
  availability_set_id = azurerm_availability_set.controllers.id

  size        = var.controller_type
-  custom_data = base64encode(data.ct_config.controller-ignitions.*.rendered[count.index])
+  custom_data = base64encode(data.ct_config.controllers.*.rendered[count.index])

  # storage
  source_image_id = var.os_image
@ -111,41 +111,22 @@ resource "azurerm_network_interface_backend_address_pool_association" "controlle
  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
 }

-# Controller Ignition configs
-data "ct_config" "controller-ignitions" {
-  count    = var.controller_count
-  content  = data.template_file.controller-configs.*.rendered[count.index]
-  strict   = true
-  snippets = var.controller_snippets
-}
-
-# Controller Fedora CoreOS configs
-data "template_file" "controller-configs" {
+# Fedora CoreOS controllers
+data "ct_config" "controllers" {
  count = var.controller_count
-
-  template = file("${path.module}/fcc/controller.yaml")
-
-  vars = {
+  content = templatefile("${path.module}/butane/controller.yaml", {
    # Cannot use cyclic dependencies on controllers or their DNS records
    etcd_name   = "etcd${count.index}"
    etcd_domain = "${var.cluster_name}-etcd${count.index}.${var.dns_zone}"
    # etcd0=https://cluster-etcd0.example.com,etcd1=https://cluster-etcd1.example.com,...
-    etcd_initial_cluster   = join(",", data.template_file.etcds.*.rendered)
+    etcd_initial_cluster = join(",", [
+      for i in range(var.controller_count) : "etcd${i}=https://${var.cluster_name}-etcd${i}.${var.dns_zone}:2380"
+    ])
    kubeconfig             = indent(10, module.bootstrap.kubeconfig-kubelet)
    ssh_authorized_key     = var.ssh_authorized_key
    cluster_dns_service_ip = cidrhost(var.service_cidr, 10)
    cluster_domain_suffix  = var.cluster_domain_suffix
-  }
+  })
+  strict   = true
+  snippets = var.controller_snippets
 }
-
-data "template_file" "etcds" {
-  count    = var.controller_count
-  template = "etcd$${index}=https://$${cluster_name}-etcd$${index}.$${dns_zone}:2380"
-
-  vars = {
-    index        = count.index
-    cluster_name = var.cluster_name
-    dns_zone     = var.dns_zone
-  }
-}
-
--- a/azure/fedora-coreos/kubernetes/lb.tf
+++ b/azure/fedora-coreos/kubernetes/lb.tf
@ -53,53 +53,45 @@ resource "azurerm_lb" "cluster" {
 }

 resource "azurerm_lb_rule" "apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name                           = "apiserver"
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "apiserver"

-  protocol                = "Tcp"
-  frontend_port           = 6443
-  backend_port            = 6443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.controller.id
-  probe_id                = azurerm_lb_probe.apiserver.id
+  protocol                 = "Tcp"
+  frontend_port            = 6443
+  backend_port             = 6443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.controller.id]
+  probe_id                 = azurerm_lb_probe.apiserver.id
 }

 resource "azurerm_lb_rule" "ingress-http" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name                           = "ingress-http"
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 80
-  backend_port            = 80
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 80
+  backend_port             = 80
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 resource "azurerm_lb_rule" "ingress-https" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name                           = "ingress-https"
  loadbalancer_id                = azurerm_lb.cluster.id
  frontend_ip_configuration_name = "ingress"
  disable_outbound_snat          = true

-  protocol                = "Tcp"
-  frontend_port           = 443
-  backend_port            = 443
-  backend_address_pool_id = azurerm_lb_backend_address_pool.worker.id
-  probe_id                = azurerm_lb_probe.ingress.id
+  protocol                 = "Tcp"
+  frontend_port            = 443
+  backend_port             = 443
+  backend_address_pool_ids = [azurerm_lb_backend_address_pool.worker.id]
+  probe_id                 = azurerm_lb_probe.ingress.id
 }

 # Worker outbound TCP/UDP SNAT
 resource "azurerm_lb_outbound_rule" "worker-outbound" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "worker"
  loadbalancer_id = azurerm_lb.cluster.id
  frontend_ip_configuration {
@ -112,16 +104,12 @@ resource "azurerm_lb_outbound_rule" "worker-outbound" {

 # Address pool of controllers
 resource "azurerm_lb_backend_address_pool" "controller" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "controller"
  loadbalancer_id = azurerm_lb.cluster.id
 }

 # Address pool of workers
 resource "azurerm_lb_backend_address_pool" "worker" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "worker"
  loadbalancer_id = azurerm_lb.cluster.id
 }
@ -130,8 +118,6 @@ resource "azurerm_lb_backend_address_pool" "worker" {

 # TCP health check for apiserver
 resource "azurerm_lb_probe" "apiserver" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "apiserver"
  loadbalancer_id = azurerm_lb.cluster.id
  protocol        = "Tcp"
@ -145,8 +131,6 @@ resource "azurerm_lb_probe" "apiserver" {

 # HTTP health check for ingress
 resource "azurerm_lb_probe" "ingress" {
-  resource_group_name = azurerm_resource_group.cluster.name
-
  name            = "ingress"
  loadbalancer_id = azurerm_lb.cluster.id
  protocol        = "Http"
--- a/azure/fedora-coreos/kubernetes/network.tf
+++ b/azure/fedora-coreos/kubernetes/network.tf
@ -41,4 +41,3 @@ resource "azurerm_subnet_network_security_group_association" "worker" {
  subnet_id                 = azurerm_subnet.worker.id
  network_security_group_id = azurerm_network_security_group.worker.id
 }
-
--- a/azure/fedora-coreos/kubernetes/outputs.tf
+++ b/azure/fedora-coreos/kubernetes/outputs.tf
@ -1,5 +1,6 @@
 output "kubeconfig-admin" {
-  value = module.bootstrap.kubeconfig-admin
+  value     = module.bootstrap.kubeconfig-admin
+  sensitive = true
 }

 # Outputs for Kubernetes Ingress
@ -32,7 +33,8 @@ output "security_group_id" {
 }

 output "kubeconfig" {
-  value = module.bootstrap.kubeconfig-kubelet
+  value     = module.bootstrap.kubeconfig-kubelet
+  sensitive = true
 }

 # Outputs for custom firewalling
@ -41,9 +43,9 @@ output "worker_security_group_name" {
  value = azurerm_network_security_group.worker.name
 }

-output "worker_address_prefix" {
-  description = "Worker network subnet CIDR address (for source/destination)"
-  value       = azurerm_subnet.worker.address_prefix
+output "worker_address_prefixes" {
+  description = "Worker network subnet CIDR addresses (for source/destination)"
+  value       = azurerm_subnet.worker.address_prefixes
 }

 # Outputs for custom load balancing
@ -57,3 +59,11 @@ output "backend_address_pool_id" {
  description = "ID of the worker backend address pool"
  value       = azurerm_lb_backend_address_pool.worker.id
 }
+
+# Outputs for debug
+
+output "assets_dist" {
+  value     = module.bootstrap.assets_dist
+  sensitive = true
+}
+
--- a/azure/fedora-coreos/kubernetes/security.tf
+++ b/azure/fedora-coreos/kubernetes/security.tf
@ -10,171 +10,171 @@ resource "azurerm_network_security_group" "controller" {
 resource "azurerm_network_security_rule" "controller-icmp" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-icmp"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "1995"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Icmp"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-icmp"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "1995"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Icmp"
+  source_port_range            = "*"
+  destination_port_range       = "*"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 resource "azurerm_network_security_rule" "controller-ssh" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-ssh"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2000"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "22"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-ssh"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2000"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "22"
+  source_address_prefix        = "*"
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 resource "azurerm_network_security_rule" "controller-etcd" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-etcd"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2005"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "2379-2380"
-  source_address_prefix       = azurerm_subnet.controller.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-etcd"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2005"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "2379-2380"
+  source_address_prefixes      = azurerm_subnet.controller.address_prefixes
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 # Allow Prometheus to scrape etcd metrics
 resource "azurerm_network_security_rule" "controller-etcd-metrics" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-etcd-metrics"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2010"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "2381"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-etcd-metrics"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2010"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "2381"
+  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 # Allow Prometheus to scrape kube-proxy metrics
 resource "azurerm_network_security_rule" "controller-kube-proxy" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-kube-proxy-metrics"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2011"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10249"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-kube-proxy-metrics"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2011"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "10249"
+  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 # Allow Prometheus to scrape kube-scheduler and kube-controller-manager metrics
 resource "azurerm_network_security_rule" "controller-kube-metrics" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-kube-metrics"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2012"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10251-10252"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-kube-metrics"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2012"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "10257-10259"
+  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 resource "azurerm_network_security_rule" "controller-apiserver" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-apiserver"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2015"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "6443"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-apiserver"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2015"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "6443"
+  source_address_prefix        = "*"
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 resource "azurerm_network_security_rule" "controller-cilium-health" {
  resource_group_name = azurerm_resource_group.cluster.name
  count               = var.networking == "cilium" ? 1 : 0

-  name                        = "allow-cilium-health"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2019"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "4240"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-cilium-health"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2019"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "4240"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 resource "azurerm_network_security_rule" "controller-vxlan" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-vxlan"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2020"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "4789"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-vxlan"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2020"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Udp"
+  source_port_range            = "*"
+  destination_port_range       = "4789"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 resource "azurerm_network_security_rule" "controller-linux-vxlan" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-linux-vxlan"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2021"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "8472"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-linux-vxlan"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2021"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Udp"
+  source_port_range            = "*"
+  destination_port_range       = "8472"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "controller-node-exporter" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-node-exporter"
-  network_security_group_name = azurerm_network_security_group.controller.name
-  priority                    = "2025"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "9100"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.controller.address_prefix
+  name                         = "allow-node-exporter"
+  network_security_group_name  = azurerm_network_security_group.controller.name
+  priority                     = "2025"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "9100"
+  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 # Allow apiserver to access kubelet's for exec, log, port-forward
@ -191,8 +191,8 @@ resource "azurerm_network_security_rule" "controller-kubelet" {
  destination_port_range      = "10250"

  # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix = azurerm_subnet.controller.address_prefix
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.controller.address_prefixes
 }

 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
@ -240,139 +240,139 @@ resource "azurerm_network_security_group" "worker" {
 resource "azurerm_network_security_rule" "worker-icmp" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-icmp"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "1995"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Icmp"
-  source_port_range           = "*"
-  destination_port_range      = "*"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-icmp"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "1995"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Icmp"
+  source_port_range            = "*"
+  destination_port_range       = "*"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 resource "azurerm_network_security_rule" "worker-ssh" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-ssh"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2000"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "22"
-  source_address_prefix       = azurerm_subnet.controller.address_prefix
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-ssh"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2000"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "22"
+  source_address_prefixes      = azurerm_subnet.controller.address_prefixes
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 resource "azurerm_network_security_rule" "worker-http" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-http"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2005"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "80"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-http"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2005"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "80"
+  source_address_prefix        = "*"
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 resource "azurerm_network_security_rule" "worker-https" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-https"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2010"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "443"
-  source_address_prefix       = "*"
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-https"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2010"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "443"
+  source_address_prefix        = "*"
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 resource "azurerm_network_security_rule" "worker-cilium-health" {
  resource_group_name = azurerm_resource_group.cluster.name
  count               = var.networking == "cilium" ? 1 : 0

-  name                        = "allow-cilium-health"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2014"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "4240"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-cilium-health"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2014"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "4240"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 resource "azurerm_network_security_rule" "worker-vxlan" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-vxlan"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2015"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "4789"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-vxlan"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2015"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Udp"
+  source_port_range            = "*"
+  destination_port_range       = "4789"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 resource "azurerm_network_security_rule" "worker-linux-vxlan" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-linux-vxlan"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2016"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Udp"
-  source_port_range           = "*"
-  destination_port_range      = "8472"
-  source_address_prefixes     = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-linux-vxlan"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2016"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Udp"
+  source_port_range            = "*"
+  destination_port_range       = "8472"
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 # Allow Prometheus to scrape node-exporter daemonset
 resource "azurerm_network_security_rule" "worker-node-exporter" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-node-exporter"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2020"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "9100"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-node-exporter"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2020"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "9100"
+  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 # Allow Prometheus to scrape kube-proxy
 resource "azurerm_network_security_rule" "worker-kube-proxy" {
  resource_group_name = azurerm_resource_group.cluster.name

-  name                        = "allow-kube-proxy"
-  network_security_group_name = azurerm_network_security_group.worker.name
-  priority                    = "2024"
-  access                      = "Allow"
-  direction                   = "Inbound"
-  protocol                    = "Tcp"
-  source_port_range           = "*"
-  destination_port_range      = "10249"
-  source_address_prefix       = azurerm_subnet.worker.address_prefix
-  destination_address_prefix  = azurerm_subnet.worker.address_prefix
+  name                         = "allow-kube-proxy"
+  network_security_group_name  = azurerm_network_security_group.worker.name
+  priority                     = "2024"
+  access                       = "Allow"
+  direction                    = "Inbound"
+  protocol                     = "Tcp"
+  source_port_range            = "*"
+  destination_port_range       = "10249"
+  source_address_prefixes      = azurerm_subnet.worker.address_prefixes
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 # Allow apiserver to access kubelet's for exec, log, port-forward
@ -389,8 +389,8 @@ resource "azurerm_network_security_rule" "worker-kubelet" {
  destination_port_range      = "10250"

  # allow Prometheus to scrape kubelet metrics too
-  source_address_prefixes    = [azurerm_subnet.controller.address_prefix, azurerm_subnet.worker.address_prefix]
-  destination_address_prefix = azurerm_subnet.worker.address_prefix
+  source_address_prefixes      = concat(azurerm_subnet.controller.address_prefixes, azurerm_subnet.worker.address_prefixes)
+  destination_address_prefixes = azurerm_subnet.worker.address_prefixes
 }

 # Override Azure AllowVNetInBound and AllowAzureLoadBalancerInBound
--- a/azure/fedora-coreos/kubernetes/ssh.tf
+++ b/azure/fedora-coreos/kubernetes/ssh.tf
@ -25,7 +25,7 @@ resource "null_resource" "copy-controller-secrets" {

  provisioner "file" {
    content     = join("\n", local.assets_bundle)
-    destination = "$HOME/assets"
+    destination = "/home/core/assets"
  }

  provisioner "remote-exec" {
--- a/azure/fedora-coreos/kubernetes/variables.tf
+++ b/azure/fedora-coreos/kubernetes/variables.tf
@ -43,7 +43,7 @@ variable "controller_type" {
 variable "worker_type" {
  type        = string
  description = "Machine type for workers (see `az vm list-skus --location centralus`)"
-  default     = "Standard_DS1_v2"
+  default     = "Standard_D2as_v5"
 }

 variable "os_image" {
@ -54,7 +54,7 @@ variable "os_image" {
 variable "disk_size" {
  type        = number
  description = "Size of the disk in GB"
-  default     = 40
+  default     = 30
 }

 variable "worker_priority" {
@ -65,13 +65,13 @@ variable "worker_priority" {

 variable "controller_snippets" {
  type        = list(string)
-  description = "Controller Fedora CoreOS Config snippets"
+  description = "Controller Butane snippets"
  default     = []
 }

 variable "worker_snippets" {
  type        = list(string)
-  description = "Worker Fedora CoreOS Config snippets"
+  description = "Worker Butane snippets"
  default     = []
 }

@ -84,8 +84,8 @@ variable "ssh_authorized_key" {

 variable "networking" {
  type        = string
-  description = "Choice of networking provider (flannel or calico)"
-  default     = "calico"
+  description = "Choice of networking provider (flannel, calico, or cilium)"
+  default     = "cilium"
 }

 variable "host_cidr" {
@ -117,8 +117,8 @@ variable "enable_reporting" {

 variable "enable_aggregation" {
  type        = bool
-  description = "Enable the Kubernetes Aggregation Layer (defaults to false)"
-  default     = false
+  description = "Enable the Kubernetes Aggregation Layer"
+  default     = true
 }

 variable "worker_node_labels" {
@ -129,15 +129,14 @@ variable "worker_node_labels" {

 # unofficial, undocumented, unsupported

-variable "asset_dir" {
-  type        = string
-  description = "Absolute path to a directory where generated assets should be placed (contains secrets)"
-  default     = ""
-}
-
 variable "cluster_domain_suffix" {
  type        = string
  description = "Queries for domains with the suffix will be answered by coredns. Default is cluster.local (e.g. foo.default.svc.cluster.local) "
  default     = "cluster.local"
 }

+variable "daemonset_tolerations" {
+  type        = list(string)
+  description = "List of additional taint keys kube-system DaemonSets should tolerate (e.g. ['custom-role', 'gpu-role'])"
+  default     = []
+}
--- a/azure/fedora-coreos/kubernetes/versions.tf
+++ b/azure/fedora-coreos/kubernetes/versions.tf
@ -1,12 +1,14 @@
 # Terraform version and plugin versions

 terraform {
-  required_version = "~> 0.12.6"
+  required_version = ">= 0.13.0, < 2.0.0"
  required_providers {
-    azurerm  = "~> 2.8"
-    ct       = "~> 0.4"
-    template = "~> 2.1"
-    null     = "~> 2.1"
+    azurerm = ">= 2.8, < 4.0"
+    null    = ">= 2.1"
+    ct = {
+      source  = "poseidon/ct"
+      version = "~> 0.9"
+    }
  }
 }

--- a/azure/fedora-coreos/kubernetes/workers/butane/worker.yaml
+++ b/azure/fedora-coreos/kubernetes/workers/butane/worker.yaml
@ -1,10 +1,12 @@
 ---
 variant: fcos
-version: 1.0.0
+version: 1.4.0
 systemd:
  units:
-    - name: docker.service
+    - name: containerd.service
      enabled: true
+    - name: docker.service
+      mask: true
    - name: wait-for-dns.service
      enabled: true
      contents: |
@ -24,8 +26,8 @@ systemd:
        Description=Kubelet (System Container)
        Wants=rpc-statd.service
        [Service]
-        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.18.6
-        ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
+        Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.26.1
+        ExecStartPre=/bin/mkdir -p /etc/cni/net.d
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/bin/mkdir -p /var/lib/calico
@ -33,64 +35,43 @@ systemd:
        ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
        ExecStartPre=-/usr/bin/podman rm kubelet
        ExecStart=/usr/bin/podman run --name kubelet \
+          --log-driver k8s-file \
          --privileged \
          --pid host \
          --network host \
+          --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
          --volume /etc/kubernetes:/etc/kubernetes:ro,z \
          --volume /usr/lib/os-release:/etc/os-release:ro \
-          --volume /etc/ssl/certs:/etc/ssl/certs:ro \
+          --volume /etc/machine-id:/etc/machine-id:ro \
          --volume /lib/modules:/lib/modules:ro \
          --volume /run:/run \
-          --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
-          --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \
-          --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
+          --volume /sys/fs/cgroup:/sys/fs/cgroup \
+          --volume /etc/selinux:/etc/selinux \
+          --volume /sys/fs/selinux:/sys/fs/selinux \
          --volume /var/lib/calico:/var/lib/calico:ro \
-          --volume /var/lib/docker:/var/lib/docker \
+          --volume /var/lib/containerd:/var/lib/containerd \
          --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
          --volume /var/log:/var/log \
          --volume /var/run/lock:/var/run/lock:z \
          --volume /opt/cni/bin:/opt/cni/bin:z \
          $${KUBELET_IMAGE} \
-          --anonymous-auth=false \
-          --authentication-token-webhook \
-          --authorization-mode=Webhook \
          --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
-          --cgroup-driver=systemd \
-          --cgroups-per-qos=true \
-          --enforce-node-allocatable=pods \
-          --client-ca-file=/etc/kubernetes/ca.crt \
-          --cluster_dns=${cluster_dns_service_ip} \
-          --cluster_domain=${cluster_domain_suffix} \
-          --cni-conf-dir=/etc/kubernetes/cni/net.d \
-          --healthz-port=0 \
+          --config=/etc/kubernetes/kubelet.yaml \
+          --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
-          --network-plugin=cni \
-          --node-labels=node.kubernetes.io/node \
          %{~ for label in split(",", node_labels) ~}
          --node-labels=${label} \
          %{~ endfor ~}
-          --pod-manifest-path=/etc/kubernetes/manifests \
-          --read-only-port=0 \
-          --rotate-certificates \
-          --volume-plugin-dir=/var/lib/kubelet/volumeplugins
+          %{~ for taint in split(",", node_taints) ~}
+          --register-with-taints=${taint} \
+          %{~ endfor ~}
+          --node-labels=node.kubernetes.io/node
        ExecStop=-/usr/bin/podman stop kubelet
        Delegate=yes
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
-    - name: delete-node.service
-      enabled: true
-      contents: |
-        [Unit]
-        Description=Delete Kubernetes node on shutdown
-        [Service]
-        Type=oneshot
-        RemainAfterExit=true
-        ExecStart=/bin/true
-        ExecStop=/bin/bash -c '/usr/bin/podman run --volume /etc/kubernetes:/etc/kubernetes:ro,z --entrypoint /usr/local/bin/kubectl quay.io/poseidon/kubelet:v1.18.6 --kubeconfig=/etc/kubernetes/kubeconfig delete node $HOSTNAME'
-        [Install]
-        WantedBy=multi-user.target
 storage:
  directories:
    - path: /etc/kubernetes
@ -100,6 +81,38 @@ storage:
      contents:
        inline: |
          ${kubeconfig}
+    - path: /etc/kubernetes/kubelet.yaml
+      mode: 0644
+      contents:
+        inline: |
+          apiVersion: kubelet.config.k8s.io/v1beta1
+          kind: KubeletConfiguration
+          authentication:
+            anonymous:
+              enabled: false
+            webhook:
+              enabled: true
+            x509:
+              clientCAFile: /etc/kubernetes/ca.crt
+          authorization:
+            mode: Webhook
+          cgroupDriver: systemd
+          clusterDNS:
+            - ${cluster_dns_service_ip}
+          clusterDomain: ${cluster_domain_suffix}
+          healthzPort: 0
+          rotateCertificates: true
+          shutdownGracePeriod: 45s
+          shutdownGracePeriodCriticalPods: 30s
+          staticPodPath: /etc/kubernetes/manifests
+          readOnlyPort: 0
+          resolvConf: /run/systemd/resolve/resolv.conf
+          volumePluginDir: /var/lib/kubelet/volumeplugins
+    - path: /etc/systemd/logind.conf.d/inhibitors.conf
+      contents:
+        inline: |
+          [Login]
+          InhibitDelayMaxSec=45s
    - path: /etc/sysctl.d/max-user-watches.conf
      contents:
        inline: |
@ -109,6 +122,13 @@ storage:
        inline: |
          net.ipv4.conf.default.rp_filter=0
          net.ipv4.conf.*.rp_filter=0
+    - path: /etc/systemd/network/50-flannel.link
+      contents:
+        inline: |
+          [Match]
+          OriginalName=flannel*
+          [Link]
+          MACAddressPolicy=none
    - path: /etc/systemd/system.conf.d/accounting.conf
      contents:
        inline: |
@ -116,10 +136,28 @@ storage:
          DefaultCPUAccounting=yes
          DefaultMemoryAccounting=yes
          DefaultBlockIOAccounting=yes
+    - path: /etc/containerd/config.toml
+      overwrite: true
+      contents:
+        inline: |
+          version = 2
+          root = "/var/lib/containerd"
+          state = "/run/containerd"
+          subreaper = true
+          oom_score = -999
+          [grpc]
+          address = "/run/containerd/containerd.sock"
+          uid = 0
+          gid = 0
+          [plugins."io.containerd.grpc.v1.cri"]
+          enable_selinux = true
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v2"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+          SystemdCgroup = true
 passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ${ssh_authorized_key}

-
--- a/azure/fedora-coreos/kubernetes/workers/variables.tf
+++ b/azure/fedora-coreos/kubernetes/workers/variables.tf
@ -41,7 +41,7 @@ variable "worker_count" {
 variable "vm_type" {
  type        = string
  description = "Machine type for instances (see `az vm list-skus --location centralus`)"
-  default     = "Standard_DS1_v2"
+  default     = "Standard_D2as_v5"
 }

 variable "os_image" {
@ -57,7 +57,7 @@ variable "priority" {

 variable "snippets" {
  type        = list(string)
-  description = "Fedora CoreOS Config snippets"
+  description = "Butane snippets"
  default     = []
 }

@ -88,6 +88,12 @@ variable "node_labels" {
  default     = []
 }

+variable "node_taints" {
+  type        = list(string)
+  description = "List of initial node taints"
+  default     = []
+}
+
 # unofficial, undocumented, unsupported

 variable "cluster_domain_suffix" {
--- a/Show More
+++ b/Show More